I think it is possible to refactor too early.
At the nuts and bolts end of design is the code itself. This final stage of the design comes in to existence as you code, it will at times be flawed, and you'll see that as the code evolves. If you refactor too early it makes it harder to change the flawed design.
For example, it's much easier to delete a single long function when you realise it's rubbish or going in the wrong direction than it is to delete a nice well-formed function and the functions it uses and the functions they use, etc., whilst ensuring you're not breaking something else that was part of the refactor.
It could be said that perhaps you should have spent more time designing, but a key element in an agile process is that coding is part of the design process and in most cases, having put some reasonable effort into design, it's better to just get on with it.
Edit In response to comments:-
Design isn't done until you've written code. We can't solve all problems in pre-coding design, the whole point behind Agile is that coding is problem solving. If the non-code design solved all problems up-front before coding there would be no need to re factor, we would simply convert the design to well factored code in one step.
Anyone remember the late 1980s and early 1990s structured design methods, the ones where you got all the problems solved in clever diagrams before you wrote a line of code?
