Do you have Antivirus running? Some AV products scan the file when they see them being closed. If the AV software still has the file open when you come to delete it, you'll see this problem. You could retry the delete after a small delay.
I've also seen this happen with search indexers.