Displaying the camera stream on a GLSurfaceView via SurfaceTexture

Question

I\'m trying to display the camera stream in a GLSurfaceView via a SurfaceTexture trasmitted to OpenGL ES 2.0 shaders.

I took inspiration from this post.

The imag

Answer 1

I have used the SurfaceTexture succesfully to draw camera frames on a custom opengl texture without using the transformation matrix provided by android.

Just try defining your indices vertices and textures the way you would do it for a normal texture draw. Like this for example.

const GLfloat Vertices[] = {0.5, -0.5, 0,
                            0.5, 0.5, 0,
                           -0.5, 0.5, 0,
                           -0.5, -0.5, 0};

const GLubyte Indices[] = { 0, 1, 2,
                            2, 3, 0 };

const GLfloat Textures[] = { 1.,0.,
                             0.,0.,
                             0.,1.,
                             1.,1. };

You should be able to use the surfacetexture the way you use a normal texture.

In case you want to perform some sort of 3D projection, this is a good post on how to generate a proper MVP matrix. Which you could use to multiply with the position in the vertex shader.

Answer 2

Since I had the same issue, even though the question is a bit old, it might be worth it to elaborate a bit about "why use the transformation matrix".

The transformation Matrix maps the particular coordinate set used by SurfaceTexture (or more likely, by the current video source streaming into SurfaceTexture...) to the standard OpenGL texture coordinates.

The reason why it should be definitely used is that, although Startibartfast answer works in one or more particular setups, and might be easy and tempting to implement, it might produce strange visualization bugs when the same program is ran on different devices, depending for instance on the video drivers implementation of each platform.

In my case, for instance, the transform matrix just flips the content upside down (I'm using SurfaceTexture in conjunction with the MediaPlayer on Nexus 7) instead of the result pointed out by Fabien R.

The correct way to use the matrix is pointed out in Fabien R edit on nov 3, 2012, which I report with some minor embellishments (like the use of st indexes), and a small correction of int/float mismatch:

attribute   vec2 uv;
varying     vec2 vTexCoord;
uniform     mat4 transformMatrix;

vTexCoord = (transformMatrix * vec4(uv, 0.0, 1.0)).st;

Hope it helps