If you want to recognize malware, you must know how it works. This means researching malware and aquirering the skill to produce malware.
- search for 29A - they wrote papers on virus
- read about rootkits (there are even books on it)
- read about reverse engineering
- read source code of malware - there's plenty of it in the web.
- learn assembler
- learn about your OS
- reverse the os-kernel
- get clam-av, check the source
I won't provide links here. They are easily found though.