Processing Android video frame by frame while recording

Question

What I am attempting to do is create an application that adds effects to videos while recording. Is there any way to have a callback method receive the frame, then apply an

Answer 1

Unfortunately, (unless I'm unaware of some other method provided by the API) the way this is done is using a direct stream to the camera and manipulating it by using some sort of Native Code to modify the stream. I've done something similar to this before when I was working on an eyetracker - So I'll tell you how it works basically.

1. Open a stream using the NDK (possibly api, depending on implementations)
2. Modify the bytes of the stream - each frame is sent as a separate packet. You have to grab each packet from the camera, and modify it. You can do a replace of colors, or you can translate. You can also use OpenGL to modify the image entirely by adding things like glass effects.
3. Flatten the images back out
4. send the image over to the view controller to be displayed.

One thing that you have to be mindful of is the load and send of the packets & images happen in about 1/30th of a second for each frame. So the code has to be extremely optimized.