Why wrapping the Data.Binary.Put monad creates a memory leak? (Part 2)

Question

As in my previous question, I\'m trying to wrap the Data.Binary.Put monad into another monad so that later I can ask it questions like \"how many bytes it\'s going to write\

Answer 1

I started playing with this and realized what the bigger problem is -- your algorithm has terrible complexity. Rather than computing the size of each child tree once, you compute it once for each time you call getSize. And you call getSize recursively. For each leaf node, getSize is called once for each time getSize is called on its parent. And getSize is called on each parent once for itself + once for each time getSize is called on any of its parents. So getsize is called at least geometrically in the depth of the tree. You need to cache the sizes to get something resembling a reasonable runtime.

That said, here's a version of the core functions that appears to run properly without a leak, although it's really crawling along for the reasons stated above:

type MyPut = S (Offset,Size) P.PutM

peal_1 :: (Monad m, Num t, Num t1) => S (t, t1) m a -> m a
peal_1 put = unS put (\o -> return) (0,0)

writeToFile :: String -> MyPut () -> IO ()
writeToFile path put =
  BL.writeFile path $ P.runPut $ (peal_1 put) >> return ()

getSize :: MyPut a -> MyPut Int
getSize x = S $ \f os -> unS (x >> getCurrentSize) f os

getCurrentOffset :: MyPut Int
getCurrentOffset = S $ \f os -> f os (fst os)

getCurrentSize :: MyPut Int
getCurrentSize = S $ \f os -> f os (snd os)

I also have to say I'm not sure if your logic is correct in general. My code preserves the current behavior while fixing the leak. I tested this by running it and your code on a cut-down data set and producing files that are bit-for-bit identical.

But for your large test data, this code wrote 6.5G before I killed it (the provided code exhausted heap well before then). I suspect but have not tested that the underlying calls in the put monad are getting run once for each call to getSize, even though the result of getSize is getting thrown away.

My proposed proper solution is posted as an answer to your other question: How do you save a tree data structure to binary file in Haskell