How to keep track of gaps in buffer and fill them

Question

I have buffers that can have things added and removed to them. The problem is that when certain sub buffers are freed there are gaps in it.

So the only way to have th