GPU PoolAllocator explodes the CPU memory

Question

I made a tensorflow model with relatively common operations (apart from a couple of tf.where and indices handling), but call it with very varying different input sh

Answer 1

This specific problem was solved some times ago by the TF team when they changed their memory allocator (see the Corresponding issue on github).

If you encounter a growth in memory during training, a common mistake is that nodes are being added to the graph during the training (TF is not numpy, unless you use eager execution). Make sure to call graph.finalize() before your training loop to ensure no nodes are added during the training process, this allows to catch many memory growth issues.