How to handle alpha in a manual “Overlay” blend operation?

Question

I\'m playing with some manual (walk-the-pixels) image processing, and I\'m recreating the standard \"overlay\" blend. I\'m looking at the \"Photoshop math\" macros here:

Answer 1

Just a guess, but I would try

resultA = 1 - (1-baseAlpha) * (1-blendAlpha)

Answer 2

After blending the base color and the blend color, mix the original base color and the color resulting from the blending using the alpha of the blend color:

vec4 baseColor = ...;
vec4 blendColor = ...;
vec4 blendedColor = blend(baseColor, blendColor);
vec4 fragmentColor = (1.0 - blendColor.a) * baseColor + blendColor.a * blendedColor;

I use this for "overlay" blending an opaque base color and a blend texture which has a lot of (semi) transparent pixels.

Answer 3

I was experimenting with this issue exactly, until I found out that the best is to have the base and the blend layer both with straight alpha, then premultiply only the result with the base alpha.