I have an ASP.NET website that will hit about 2gb physical memory used within about 3-4 days, which to me sounds really bad. At the moment, I have configured IIS to restart
Use a memory profiler (either on a memory dump, or on your live environment) when your memory exceeds that 500 MB to get a clue which objects take up all that space.
When profiling on your development environment you might see one particular objecttype only grow when surfing around on your site whilst others remain globally the same.