How do you capture current frame from a MediaElement in WinRT (8.1)?

Question

I am trying to implement a screenshot functionality in a WinRT app that shows Video via a MediaElement. I have the following code, it saves a screenshot that's the size of the MediaElement but the image is empty (completely black). Tried with various types of Media files. If I do a Win Key + Vol Down on Surface RT, the screen shot includes the Media frame content, but if I use the following code, it's blackness all around :(

private async Task SaveCurrentFrame()
{
 RenderTargetBitmap renderTargetBitmap = new RenderTargetBitmap();
 await renderTargetBitmap.RenderAsync(Player);
 var pixelBuffer = await renderTargetBitmap.GetPixelsAsync();
 MultimediaItem currentItem = (MultimediaItem)this.DefaultViewModel["Group"];
 StorageFolder currentFolder = Windows.Storage.ApplicationData.Current.LocalFolder;
 var saveFile = await currentFolder.CreateFileAsync(currentItem.UniqueId + ".png", CreationCollisionOption.ReplaceExisting);
 if (saveFile == null)
    return;
 // Encode the image to the selected file on disk
 using (var fileStream = await saveFile.OpenAsync(FileAccessMode.ReadWrite))
 {
    var encoder = await BitmapEncoder.CreateAsync(BitmapEncoder.PngEncoderId, fileStream);

    encoder.SetPixelData(
        BitmapPixelFormat.Bgra8,
        BitmapAlphaMode.Ignore,
        (uint)renderTargetBitmap.PixelWidth,
        (uint)renderTargetBitmap.PixelHeight,
        DisplayInformation.GetForCurrentView().LogicalDpi,
        DisplayInformation.GetForCurrentView().LogicalDpi,
        pixelBuffer.ToArray());

    await encoder.FlushAsync();
 }
}

Here MultimediaItem is my View Model class that among other things has a UniqueId property that's a string.

'Player' is the name of the Media Element.

Is there anything wrong with the code or this approach is wrong and I've to get in the trenches with C++?

P.S. I am interested in the WinRT API only.

Update 1 Looks like RenderTargetBitmap doesn't support this, the MSDN documentation clarifies it http://msdn.microsoft.com/en-us/library/windows/apps/windows.ui.xaml.media.imaging.rendertargetbitmap . I'll appreciate any pointers on how to do it using DirectX C++. This is a major task for me so I'll crack this one way or the other and report back with the solution.

Answer 1

Yes, it is possible - little bit tricky, but working well.

You dont use mediaElement, but StorageFile itself. You need to create writableBitmap with help of Windows.Media.Editing namespace.

Works in UWP (Windows 10)

This is complete example with file picking and getting video resolution and saving image to Picture Library

        TimeSpan timeOfFrame = new TimeSpan(0, 0, 1);//one sec

        //pick mp4 file
        var picker = new Windows.Storage.Pickers.FileOpenPicker();
        picker.SuggestedStartLocation = Windows.Storage.Pickers.PickerLocationId.VideosLibrary;
        picker.FileTypeFilter.Add(".mp4");
        StorageFile pickedFile = await picker.PickSingleFileAsync();
        if (pickedFile == null)
        {
            return;
        }
        ///


        //Get video resolution
        List<string> encodingPropertiesToRetrieve = new List<string>();
        encodingPropertiesToRetrieve.Add("System.Video.FrameHeight");
        encodingPropertiesToRetrieve.Add("System.Video.FrameWidth");
        IDictionary<string, object> encodingProperties = await pickedFile.Properties.RetrievePropertiesAsync(encodingPropertiesToRetrieve);
        uint frameHeight = (uint)encodingProperties["System.Video.FrameHeight"];
        uint frameWidth = (uint)encodingProperties["System.Video.FrameWidth"];
        ///


        //Use Windows.Media.Editing to get ImageStream
        var clip = await MediaClip.CreateFromFileAsync(pickedFile);
        var composition = new MediaComposition();
        composition.Clips.Add(clip);

        var imageStream = await composition.GetThumbnailAsync(timeOfFrame, (int)frameWidth, (int)frameHeight, VideoFramePrecision.NearestFrame);
        ///


        //generate bitmap 
        var writableBitmap = new WriteableBitmap((int)frameWidth, (int)frameHeight);
        writableBitmap.SetSource(imageStream);


        //generate some random name for file in PicturesLibrary
        var saveAsTarget = await KnownFolders.PicturesLibrary.CreateFileAsync("IMG" + Guid.NewGuid().ToString().Substring(0, 4) + ".jpg");


        //get stream from bitmap
        Stream stream = writableBitmap.PixelBuffer.AsStream();
        byte[] pixels = new byte[(uint)stream.Length];
        await stream.ReadAsync(pixels, 0, pixels.Length);

        using (var writeStream = await saveAsTarget.OpenAsync(FileAccessMode.ReadWrite))
        {
            var encoder = await BitmapEncoder.CreateAsync(BitmapEncoder.JpegEncoderId, writeStream);
            encoder.SetPixelData(
                BitmapPixelFormat.Bgra8,
                BitmapAlphaMode.Premultiplied,
                (uint)writableBitmap.PixelWidth,
                (uint)writableBitmap.PixelHeight,
                96,
                96,
                pixels);
            await encoder.FlushAsync();

            using (var outputStream = writeStream.GetOutputStreamAt(0))
            {
                await outputStream.FlushAsync();
            }
        }

Yeah...I spent lot of hours by this

Answer 2

Ok I have managed to get making snapshot from MediaElement on button press to work.

I am passing MediaStreamSource object to MediaElement using SetMediaStreamSource method. MediaStreamSource has event SampleRequested which is fired basicly everytime new frame is drawn. Then using boolean I control when to create bitmap

private async void MediaStream_SampleRequested(MediaStreamSource sender, MediaStreamSourceSampleRequestedEventArgs args)
{
    if (!takeSnapshot)
    {
        return;
    }

    takeSnapshot = false;
    Task.Run(() => DecodeAndSaveVideoFrame(args.Request.Sample));
}

After that what is left is to decode compressed image and convert it to WriteableBitmap. The image is (or at least was in my case) in YUV fromat. You can get the byte array using

byte[] yvuArray = sample.Buffer.ToArray();

and then get data from this array and convert it to RGB. Unfortunetly I cannot post entire code but I'm gonna give you a few more hints:

YUV to RGB wiki here you have wiki describing how does YUV to RGB conversion works.

Here I found python project which solution I have adapted (and works perfectly). To be more precise you have to analize how method NV12Converter works.

The last thing is to change takeSnapshot boolean to true after pressing button or doing other activity :).