Generating Smooth Normals from active Vertex Array

Question

I'm attempting to hack and modify several rendering features of an old opengl fixed pipeline game, by hooking into OpenGl calls, and my current mission is to implement shader lighting. I've already created an appropriate shader program that lights most of my objects correctly, but this game's terrain is drawn with no normal data provided.

The game calls:

void glVertexPointer(GLint size, GLenum type, GLsizei stride, const GLvoid * pointer);

and

void glDrawElements(GLenum mode, GLsizei count, GLenum type, const GLvoid * indices);`

to define and draw the terrain, thus I have these functions both hooked, and I hope to loop through the given vertex array at the pointer, and calculate normals for each surface, on either every DrawElements call or VertexPointer call, but I'm having trouble coming up with an approach to do so - specifically, how to read, iterate over, and understand the data at the pointer. In this case, the usual parameters for the glVertexPointer calls are size = 3, type = GL_float, stride = 16, pointer = some pointer. Hooking glVertexPointer, I don't know how I could iterate through the pointer and grab all the vertices for the mesh, considering I don't know the total count of all the vertices, nor do I understand how the data is structured at the pointer given the stride - and similarly how i should structure the normal array

Would it be a better idea to try to calculate the normals in drawelements for each specified index in the indice array?

Answer 1

Depending on your vertex array building procedure, indices would be the only relevant information for building your normals.

Difining normal average for one vertex is simple if you add a normal field in your vertex array, and sum all the normal calculations parsing your indices array.

You have than to divide each normal sum by the number of repetition in indices, count that you can save in a temporary array following vertex indices (incremented each time a normal is added to the vertex)

so to be more clear:

Vertex[vertexCount]: {Pos,Normal}
normalCount[vertexCount]: int count

Indices[indecesCount]: int vertexIndex

You may have 6 normals per vertex so add a temporary array of normal array to averrage those for each vertex:

NormalTemp[vertexCount][6] {x,y,z}

than parsing your indice array (if it's triangle):

for i=0 to indicesCount step 3
   for each triangle top (t from 0 to 2)
       NormalTemp[indices[i + t]][normalCount[indices[i+t]]+1] = normal calculation with cross product of vectors ending with other summits or this triangle
       normalCount[indices[i+t]]++

than you have to divide your sums by the count

for i=0 to vertexCount step 1
    for j=0 to NormalCount[i] step 1
        sum += NormalTemp[i][j]
    normal[i] = sum / normacount[i]

Answer 2

While I like and have voted up the j-p's answer I would still like to point out that you could get away with calculating one normal per face and just using for all 3 vertices. It would be faster, and easier, and sometimes even more accurate.