It really seems to be memory leaking. Probably, some of the UI elements in the DataTemplate keep references to other objects that should stay alive even when the UI element is destroyed.
There could be some memory leaks with Image control. Try removing it from the template and see the result. Also, are you subscribing to any events in controls' Loaded events or something like that?
Just some guesses though... As people already said here you might really want to look at your app with performance and memory profilers.