I faced the same problem, but in correlation to Interface Builder and Storyboard access.
After some thinking, it becomes clear that after Storyboard file in my project enlarged to some amount (currently about 1.5MB on disk), XCode and IB have problem.
While changing text in some (e.g. IBLabel) object, it needs about 0.5 seconds for each letter, what gives 3 seconds for a change of 6 character word. That slows me a lot.
XCode restart helps and releases all occupied memory...until next working session.
I have strong impression that IB makes undo buffer after each change (e.g. change single letter) because after each single change, memory footprint enlarges for about 20 MB (probably Storyboard size in RAM, on disk is 1.5 MB).
And after XCode restart, undo buffer has gone, and some memory has accordingly released.
I would like to know a way to limit undo buffer size, and test it with that smaller size.
Configuration is: Mac Mini, 4 GB RAM, OS X 10.8.3, XCode 4.6, XCode 4.6.1, XCode 4.6.2
I am currently back to XCode 4.5 and situation seems a little bit better but problem is still here. I suppose this version has less features than 4.6 and according that difference it uses less memory.
Ok, seems like XCode 4.5 actually CAN release some of memory that Interface Builder allocates.
Update on May 21. 2013:
I upgraded from 4GB to Mac Mini 2011 8GB RAM and did not notice problem yet.