At the risk of being glib, you have a memory leak. Why not try a tool like ANTS* to track it down. They have a free trial, I've never used it but it has a good reputation.
*Other profiling tools are available.
If you don't want to get to grips with another tool, you can try something like incrementing a static member every time a class is created and decrementing it every time an instance is disposed. This will help you track down instances that are not be destroyed properly.