I replaced
av_seek_frame(format_context, stream_index, request_timestamp, 0);
with
avformat_seek_file(format_context, stream_index, INT64_MIN, request_timestamp, INT64_MAX, 0);
and suddenly I get reasonable output. Great.
And it only took a day in almost complete documentation darkness. :/