FFMPEG: Overlaying one video on another one, and making black pixels transparent

Question

I\'m trying to use FFMPEG to create a video with one video overlayed on top another.

I have 2 MP4s. I need to make all BLACK pixels

Answer 1

You will not be able to get a "replace black pixels" approach to work properly. What you actually want is a foreground video with a real alpha channel that can be manipulated and tested before doing an overlay on a background. For an extended example that describes the problems, please take a look at my blog post on the subject. When using FFMPEG, an easy way to import alpha channel video is to use Quicktime with the Animation codec video at 32 BPP.

Answer 2

The notional way to do this is to chroma-key the black out and then overlay, But as @MoDJ said, this likely won't produce satisfactory results. Neither will the method I suggest below, but it's worth a try.

ffmpeg -i 1.mp4 -i 2.mp4 -filter_complex
"[1]split[m][a];
 [a]geq='if(gt(lum(X,Y),16),255,0)',hue=s=0[al];
 [m][al]alphamerge[ovr];
 [0][ovr]overlay"
output.mp4

Above, I duplicate the overlay video stream, then use the geq filter to manipulate the luma values so that any pixel with luma greater than 16 (i.e. not pure black) has its luma set to white, else zero. Since I haven't provided expressions for the two color channels, geq falls back on the luma expression. We don't want that, so I use the hue filter to nullify those channels. Then I use the alphamerge filter to merge this as an alpha channel with the first copy of the overlay video. Then, the overlay. Like I said, this may not produce satisfactory results. You can tweak the value 16 in the geq filter to change the black threshold. Suggested range is 16-24 for limited-range (Y: 16-235) video files.