So, there's a few things going on.
- avpicture_fill() does not set frame->width/height/format. You have to set these values yourself.
- avpicture_get_size() and avpicture_fill() do not guarantee alignment. The underlying functions called in these wrappers (e.g. av_image_get_buffer_size() or av_image_fill_arrays()) are called with align=1, so there's no buffer alignment between lines. If you want alignment (you do), you either have to call the underlying functions directly with a different align setting, or call avcodec_align_dimensions2() on the width/height and provide aligned width/height to the avpicture_*() functions. If you do that, you can also consider using avpicture_alloc() instead of avpicture_get_size() + av_malloc() + avpicture_fill().
I think if you follow these two suggestions, you'll find that the rescaling works as expected, gives no warnings and has correct output. The quality may not be great because you're trying to do bilinear scaling. Most people use bicubic scaling (SWS_BICUBIC).