I had the same problem using Mac's Terminal. I think it is an issue with it. Once I upgraded to Lion I don't have the problem of scrolling in vim. Please consider using iTerm2. It will solve the problem of scrolling and gives you the 256 term colors.