I am still not getting exactly what you want to achieve.
Do you want to i.e. compare the different smiley images you get with the happy or sad reference smiley picture and see whether one of the given references is shown in that picture or do you really want to extract whether the smiley/person on that image is smiling?
Latter is not so easy to achieve. Most attempts to get emotions of people on pictures extract the different features there can be seen in the face, i.e. width of mouth, position and "openness" of eyes, position and angle of eyebrows and so on... you would need to implement methods to get these features (there are several methods for that) and classify the feature-vector to your given test results to get a best match on which emotion it is... But all this is really not done easily...
If you really would like to compare how similar two images are or whether one image is in the other (i.e. whether there is that yellow smiley in an image), you could try a template match on the two images...
