Convert OpenCV Mat to Texture2D?

Question

Meta Context:

I\'m currently working on a game that utilizes opencv as a substitute for ordinary inputs (keyboard, mouse, etc...). I\'m using Unity3

Answer 1

Use SetPixels32 instead of LoadRawTextureData. Instead of returning the array data from C++, do that from C#. Create Color32 array and pin it in c# with GCHandle.Alloc, send the address of the pinned Color32 array to C++, use cv::resize to resize the cv::Mat to match the size of pixels sent from C#. You must do this step or expect some error or issues.

Finally, convert cv::Mat from RGB to ARGB then use std::memcpy to update the array from C++. The SetPixels32 function can then be used to load that updated Color32 array into Texture2D. This is how I do it and it has been working for me without any issues. There might be other better ways to do it but I have never found one.

C++:

cv::Mat _currentFrame;

void GetRawImageBytes(unsigned char* data, int width, int height)
{
   //Resize Mat to match the array passed to it from C#
    cv::Mat resizedMat(height, width, _currentFrame.type());
    cv::resize(_currentFrame, resizedMat, resizedMat.size(), cv::INTER_CUBIC);

    //You may not need this line. Depends on what you are doing
    cv::imshow("Nicolas", resizedMat);

    //Convert from RGB to ARGB 
    cv::Mat argb_img;
    cv::cvtColor(resizedMat, argb_img, CV_RGB2BGRA);
    std::vector<cv::Mat> bgra;
    cv::split(argb_img, bgra);
    std::swap(bgra[0], bgra[3]);
    std::swap(bgra[1], bgra[2]);
    std::memcpy(data, argb_img.data, argb_img.total() * argb_img.elemSize());
}

---

C#:

Attach to any GameObject with a Renderer and you should see the cv::Mat displayed and updated on that Object every frame. Code is commented if confused:

using System;
using System.Runtime.InteropServices;
using UnityEngine;

public class Test : MonoBehaviour
{
    [DllImport("ImageInputInterface")]
    private static extern void GetRawImageBytes(IntPtr data, int width, int height);

    private Texture2D tex;
    private Color32[] pixel32;

    private GCHandle pixelHandle;
    private IntPtr pixelPtr;

    void Start()
    {
        InitTexture();
        gameObject.GetComponent<Renderer>().material.mainTexture = tex;
    }


    void Update()
    {
        MatToTexture2D();
    }


    void InitTexture()
    {
        tex = new Texture2D(512, 512, TextureFormat.ARGB32, false);
        pixel32 = tex.GetPixels32();
        //Pin pixel32 array
        pixelHandle = GCHandle.Alloc(pixel32, GCHandleType.Pinned);
        //Get the pinned address
        pixelPtr = pixelHandle.AddrOfPinnedObject();
    }

    void MatToTexture2D()
    {
        //Convert Mat to Texture2D
        GetRawImageBytes(pixelPtr, tex.width, tex.height);
        //Update the Texture2D with array updated in C++
        tex.SetPixels32(pixel32);
        tex.Apply();
    }

    void OnApplicationQuit()
    {
        //Free handle
        pixelHandle.Free();
    }
}