Rotate Normals in Shader

Question

I have a scene with several models with individual positions and rotations. Given normals, the shaders apply simple bidirectional lighting to each pixel.

That is my vertex shader.

#version 150

in vec3 position;
in vec3 normal;
in vec2 texcoord;

out vec3 f_normal;
out vec2 f_texcoord;

uniform mat4 model;
uniform mat4 view;
uniform mat4 proj;

void main()
{
    mat4 mvp = proj * view * model;

    f_normal   = normal;
    f_texcoord = texcoord;

    gl_Position = mvp * vec4(position, 1.0);
}

And here is the fragment shader.

#version 150

in vec3 f_normal;
in vec2 f_texcoord;

uniform sampler2D tex;

vec3 sun = vec3(0.5, 1.0, 1.5);

void main()
{
    vec3 light = max(0.0, dot(normalize(f_normal), normalize(sun)));

    gl_FragColor = texture(tex, f_texcoord) * vec4(light, 1.0);
}

For objects without rotation this works fine. But for rotated models the lighting is rotated as well which, of course, shouldn't be the case.

The reason for that is that the normals are not rotated. I already tried f_normal = model * normal; but this applies both rotation and transformation to the normals.

So how can I rotate the normals in the vertex shader before I send them to the fragment shader for lighting? What is a common approach?

Answer 1

You do need to transform the normals, by the upper 3 rows/cols of the model-view-projection matrix. (If you are performing any scaling though, you need to use the inverse transpose of this matrix. See this article).

mat3 normalMatrix = mat3(mvp);
normalMatrix = inverse(normalMatrix);
normalMatrix = transpose(normalMatrix);
f_normal = normalize(normal * normalMatrix);
// You should also send your tranformed position to the fragment shader
f_position = vec3(mvp * vec4(position, 1.0));

In your fragment shader, you need to calculate the distance from your light source to your fragment and normalize. Find the dot product of the normal and the light vector and multiply this by the light color.

vec3 light = normalize(sun - f_position);
light = max(dot(f_normal, light), 0.0) * vec3(1.0, 1.0, 1.0);
gl_FragColor = texture(tex, f_texcoord) * vec4(light, 1.0);

There is definitely room for optimization in my code.

I recommend this book OpenGL 4.0 Shading Language Cookbook.

Answer 2

The following solution works with my models, but I don't have a very deep understanding of the maths behind it. This is my source: Adding Depth and Realism - Surface Normals

The transform you need to apply to your normal vectors is denoted by: N' = N * (M^-1)^T

Basically, this means that you multiply your normals (N) by the inverse-transpose of your ModelView matrix (M). If your M matrix is 4x4, you should just use the resulting 3x3 upper-left quadrant of (M^-1)^T for the normal multiplication.

Again, this works for me but I can't explain the maths too well.