Refactor whenever you have the opportunity to:
- introduce the appropriate design patterns in lieu of spaghetti code
- reduce code complexity and increase readability
- remove redundant code by introducing a single method call
- extricate UI from business logic
etc, etc.
EDIT:
Obviously, the decision to refactor should take into account time, potential pay-off, other responsibilities, etc.