Inconsistent sample time / presentation time during video decoding

Question

I\'m writing an APP that can encode video by camera input and process video by decode-edit-encode steps. For the camera, I use the Camera class rather than Intent to configu

Answer 1

There is no guarantee that MediaCodec handles presentation time stamps correctly before Android 4.3. This is because the CTS tests that confirm the PTS behavior were not added until then.

I do recall that there were problems with the timestamp handling in the AVC codecs from certain vendors. I don't remember the details offhand, but if you run the buffer-to-buffer and buffer-to-surface tests from EncodeDecodeTest on a variety of 4.1/4.2 devices you'll turn up some failures. (You'd need to strip out the surface-to-surface tests, of course.)

Your timestamp handling code looks fine. The timestamp isn't part of the H.264 stream, so it's really just being forwarded through the codec as metadata, and you seem to be picking it up and forwarding it on in all the right places. The bottom line is, if you're passing valid PTS values in and getting good video but garbage PTS values, something in the codec is mishandling them.

You'll need to work around it by passing the values separately, or -- if the input frame rate is always regular -- trivially computing it. In theory the encoder can reorder frames, so the order in which you pass time stamps into the encoder might not be the same order in which they come out... but since you know the timestamps were ascending when you made the movie, you should be able to just sort them if this were a problem in practice.

On a different note, delays in the system will cause the "wobble" you're seeing in the timestamp values if you grab System.nanoTime() when the frame arrives in the app. You can do a bit better in Android 4.3 with Surface input because the SurfaceTexture holds a timestamp that is set much closer to when the frame was captured. (I know that's not useful for your current efforts, but wanted to give some hope for the future.)