how to perform stable eye corner detection?

Question

For those who find it too long, just read the bold lines.

My project of gaze estimation based screen cursor moving HCI is now dependent on one last thin

Answer 1

I think there is an easy way to help!

It looks as though you are considering each eye in isolation. What I suggest you do is to combine your data for both eyes, and also use facial geometry. I will illustrate my suggestions with a picture that some people may recognise (it is not really the best example, as its a painting, and her face is a bit off centre, but it is certainly the funniest..)

It seems you have relible estimates for the pupil position for both eyes, and providing the face is looking fairly straight on at the camera (facial rotations perpendicular to the screen will be ok using this method), we know that the corners of the eyes (from now on just 'corners') will lie on (or near to) the line that passes through the pupils of both eyes (red dotted line).

We know the distance between the pupils, a, and we know that the ratio between this distance, and the distance across one eye (corner to corner), b, is fixed for an individual, and will not change much across the adult population (may differ between sexes).

ie. a / b = constant.

Therefore we can deduce b, independent of the subjects distance from the camera, knowing only a.

Using this information we can construct threshold boxes for each eye corner (dotted boxes, in detail, labelled 1, 2, 3, 4). Each box is b by c (eye height, again determinable through the same fixed ratio principle) and lies parrallel to the pupil axis. The centre edge of each box is pinned to the centre of the pupil, and moves with it. We know each corner will always be in its very own threshold box!

Now, of course the trouble is the pupils move about, and so do our threshold boxes... but we've massively narrowed down the field this way, because we can confidently discard ALL estimate eye positions (from Harris or GFTT or anything) falling outside of these boxes (provided we are confident about our pupil detection).

• If we have high confidence in just one corner position we can extrapolate and deduce all the other corner postions just from geometry! (for both eyes!).

• If there is doubt between multiple corner positions we can use knowledge of other corners (from either eye) to resolve it probabilistically linking their positions, making a best guess. ie. do any pair of estimates (within their boxes of course) lie b apart and parallel to the pupil axis.

• If you can get general 'eye' positions that do not move when the pupil moves around (or in fact any facial feature on the same plane), this is massively useful and allows you to determine the corners positions geometrically.

I hope this might help you find the elusive d (pupil displacement from center of eye).

Answer 2

i changed this

if ( x<colorImage.width/5 or x>((colorImage.width/4)*3) ) and (y>40 and y<70):

to this:

if ( x<(w/5) or x>((w/4)*3) ) and (y>int(h*0.45) and y<int(h*0.65)):

because earlier i was just manually looking at pixel values beyond which i my windows where corners could be found with the highest probability. But then afterwards i realised, lets make it general, so i made a horizontal window of 45 to 65 pc of the Y range, and 1/5th to 3/4ths for X range, because that's the usual area within which the corners are.

I'm sorry guys for replying late, i was busy with the later part of the project - gaze estimation. And i'm gonna post a question about it, i'm stuck in it.

by the way, here are few pictures of eye corners and pupil detected in my eye: (enlarged to 100x100)

Hope this will be useful for others beginning in this area.

Answer 3

Have you tried sclera segmentation?

You might be able to do with the 2 corners of sclera as well, and this might be easier because you already have a decent pupil detection working, sclera is the brighter region surrounding the pupil.