Reducing memory usage in an extended Mathematica session

Question

I\'m doing some rather long computations, which can easily span a few days. In the course of these computations, sometimes Mathematica will run out of memory. To this end, I\'ve

Answer 1

Have you tried to evaluate $HistoryLength=0; in all subkernels and as well as in the master kernel? History tracking is the most common source for going out of memory.

Have you tried do not use slow and memory-consuming Export and use fast and efficient Put instead?

It is not clear from your post where you evaluate ClearSystemCache[] - in the master kernel or in subkernels? It looks like you evaluate it in the master kernel only. Try to evaluate it in all subkernels too before each iteration.

Answer 2

Since you are using Module extensively, I think you may be interested in knowing this bug with non-deleting temporary Module variables.

Example (non-deleting unlinked temporary variables with their definitions):

In[1]:= $HistoryLength=0;
a[b_]:=Module[{c,d},d:=9;d/;b===1];
Length@Names[$Context<>"*"]

Out[3]= 6

In[4]:= lst=Table[a[1],{1000}];
Length@Names[$Context<>"*"]

Out[5]= 1007

In[6]:= lst=.
Length@Names[$Context<>"*"]

Out[7]= 1007

In[8]:= Definition@d$999

Out[8]= Attributes[d$999]={Temporary}

d$999:=9

Note that in the above code I set $HistoryLength = 0; to stress this buggy behavior of Module. If you do not do this, temporary variables can still be linked from history variables (In and Out) and will not be removed with their definitions due to this reason in more broad set of cases (it is not a bug but a feature, as Leonid mentioned).

UPDATE: Just for the record. There is another old bug with non-deleting unreferenced Module variables after Part assignments to them in v.5.2 which is not completely fixed even in version 7.0.1:

In[1]:= $HistoryLength=0;$Version
Module[{L=Array[0&,10^7]},L[[#]]++&/@Range[100];];
Names["L$*"]
ByteCount@Symbol@#&/@Names["L$*"]
Out[1]= 7.0 for Microsoft Windows (32-bit) (February 18, 2009)
Out[3]= {L$111}
Out[4]= {40000084}