I am a strong believer in constant refactoring. There is no reason to wait until some specific time to start refactoring.
Anytime you see something that should be done better, Refactor.
Just keep this in my mind. I know a developer (a pure genius) who refactors so much (he is so smart he can always find a better way) he never finishes a project.
