We found small refactorings are best done while we were working on a bit of code - do what's required, preferably paired.
For bigger things, we had a Technical Debt section on the wall - if you spotted something and didn't have the time to address it, or it was going to take some discussion to solve, you'd add it to the wall and they would be scheduled for future iterations (or when free time cropped up).
