Profiling memory usage in Mathematica

Question

Is there any way to profile the mathkernel memory usage (down to individual variables) other than paying $$$ for their Eclipse plugin (mathematica workbench, iirc)?

Answer 1

Michael Pilat's answer is a good one, and MemoryInUse and MaxMemoryUsed are probably the best tools you have. ByteCount is rarely all that helpful because what it measures can be a huge overestimate because it ignores shared subexpressions, and it often ignores memory that isn't directly accessible through Mathematica functions, which is often a major component of memory usage.

One thing you can do in some circumstances is use the Share function, which forces subexpressions to be shared when possible. In some circumstances, this can save you tens or even hundreds of magabytes. You can tell how well it's working by using MemoryInUse before and after you use Share.

Also, some innocuous-seeming things can cause Mathematica to use a whole lot more memory than you expect. Contiguous arrays of machine reals (and only machine reals) can be allocated as so-called "packed" arrays, much the way they would be allocated by C or Fortran. However, if you have a mix of machine reals and other structures (including symbols) in an array, everything has to be "boxed", and the array becomes an array of pointers, which can add a lot of overhead.

Answer 2

One way is to automatize restarting of kernel when it goes out of memory. You can execute your memory-consuming code in a slave kernel while the master kernel only takes the result of computation and controls memory usage.

Answer 3

One thing a lot of users don't realize is that it takes memory to store all your inputs and outputs in the In and Out symbols, regardless of whether or not you assign an output to a variable. Out is also aliased as %, where % is the previous output, %% is the second-to-last, etc. %123 is equivalent to Out[123].

If you don't have a habit of using %, or only use it to a few levels deep, set $HistoryLength to 0 or a small positive integer, to keep only the last few (or no) outputs around in Out.

You might also want to look at the functions MaxMemoryUsed and MemoryInUse.

Of course, the $HistoryLength issue may or not be your problem, but you haven't shared what your actual evaluation is. If you're able to post it, perhaps someone will be able to shed more light on why it's so memory-intensive.

Answer 4

Here is my solution for profiling of memory usage:

myByteCount[symbolName_String] := 
  Replace[ToHeldExpression[symbolName], 
   Hold[x__] :> 
    If[MemberQ[Attributes[x], Protected | ReadProtected], 
     Sequence @@ {}, {ByteCount[
       Through[{OwnValues, DownValues, UpValues, SubValues, 
          DefaultValues, FormatValues, NValues}[Unevaluated@x, 
         Sort -> False]]], symbolName}]];

With[{listing = myByteCount /@ Names[]},
 Labeled[Grid[Reverse@Take[Sort[listing], -100], Frame -> True, 
   Alignment -> Left], 
  Column[{Style[
     "ByteCount for symbols without attributes Protected and \
ReadProtected in all contexts", 16, FontFamily -> "Times"], 
    Style[Row@{"Total: ", Total[listing[[All, 1]]], " bytes for ", 
       Length[listing], " symbols"}, Bold]}, Center, 1.5], Top]]

Evaluation the above gives the following table: