real memory vs profiled memory python

问题

Using memory_profiler to aid in project that is requiring freeing up some memory at various points. The development environment is OS X snow leopard.

The profiled memory, as shown below, is peaking at around 414.699 MiB, but the Activity Monitor is showing the process peaking at nearly twice that (more than 900 MB).

    Line #    Mem usage    Increment   Line Contents
================================================
    24   20.441 MiB    0.000 MiB   @profile
    25                             def do_work():
    26                                 "Call each function in order"
    27   20.445 MiB    0.004 MiB       x = audio.AudioQuantumList() 
    28  137.098 MiB  116.652 MiB       audiofile = make_objects("/Users/path/audio/Track01.mp3")
    29  295.480 MiB  158.383 MiB       audiofile2 = make_objects("/Users/path/audio/Track02.mp3")
    30  414.699 MiB  119.219 MiB       audiofile3 = make_objects("/Users/path/audio/Track03.mp3")
    31  414.699 MiB    0.000 MiB       x = add_to_list(audiofile, x)
    32  417.426 MiB    2.727 MiB       audiofile = clear_memory(audiofile)  
    33  417.426 MiB    0.000 MiB       gc.collect()
    34  417.426 MiB    0.000 MiB       x = add_to_list(audiofile2, x)
    35  425.047 MiB    7.621 MiB       audiofile2 = clear_memory(audiofile2)  
    36  285.344 MiB -139.703 MiB       gc.collect()
    37  285.344 MiB    0.000 MiB       x = add_to_list(audiofile3, x)
    38  340.082 MiB   54.738 MiB       audiofile3 = clear_memory(audiofile3)  
    39  339.582 MiB   -0.500 MiB       gc.collect()

Is it normal for memory_profiler to show half of the memory that's actually being used by the CPU(s)? And is that in fact what is occurring here?

Also note that when gc.collect is not explicitly called, the -139.703 MiB freed at line 36 become:

================================================
35  374.895 MiB  -45.617 MiB       audiofile2 = clear_memory(audiofile2)

回答1:

In the line-by-line report, memory_profiler measures the memory usage after the execution of each line. In the memory peaks inside a function, e.g inside make_objects but the memory is released before the function returns, then memory_profiler will not report that usage.

Workarounds include decorating also the nested functions (e.g. make_objects) or using mprof (distributed with memory_profiler) to report memory usage as a function of time.

来源：https://stackoverflow.com/questions/25633845/real-memory-vs-profiled-memory-python