I actually think the opposite.
The earlier you start thinking about whether or not your design needs refactoring, the better. Refactor constantly, so it's never a large issue.
I've also found that the more I refactor early on, the better I've gotten about writing code more cleanly up front. I tend to create fewer large methods, and have fewer problems.
However, if you find yourself "refactoring" yourself into a corner, I'd expect that is more a matter of lack of initial design or lack of planning for the scope of use of a class. Try writing out how you want to use the class or framework before you start writing the code - it may help you avoid that issue. This is also I think one advantage to test driven design - it helps you force yourself to look at using your object before it's written.
Remember, refactoring technically should NEVER lock you into a corner - it's about reworking the internals without changing how a class is used. If your trapping yourself by refactoring, it means your initial design was flawed.
Chances are you'll find that, over time, this issue gets better and better. Your class and framework design will probably end up more flexible.
