I am writing a little program that creates an index of all files on my directories. It basically iterates over each file on the disk and stores it into a searchable database
Try using git. Version control software is geared towards this kind of problem, and git has a good reputation for speed; it's specifically designed for working fast with local files. 'git diff --name-status' would get you what you want I think.