Total memory used by Python process?

Question

Is there a way for a Python program to determine how much memory it\'s currently using? I\'ve seen discussions about memory usage for a single object, but what I need is tot

Answer 1

Even easier to use than /proc/self/status: /proc/self/statm. It's just a space delimited list of several statistics. I haven't been able to tell if both files are always present.

/proc/[pid]/statm

Provides information about memory usage, measured in pages. The columns are:

• size (1) total program size (same as VmSize in /proc/[pid]/status)
• resident (2) resident set size (same as VmRSS in /proc/[pid]/status)
• shared (3) number of resident shared pages (i.e., backed by a file) (same as RssFile+RssShmem in /proc/[pid]/status)
• text (4) text (code)
• lib (5) library (unused since Linux 2.6; always 0)
• data (6) data + stack
• dt (7) dirty pages (unused since Linux 2.6; always 0)

Here's a simple example:

from pathlib import Path
from resource import getpagesize

PAGESIZE = getpagesize()
PATH = Path('/proc/self/statm')


def get_resident_set_size() -> int:
    """Return the current resident set size in bytes."""
    # statm columns are: size resident shared text lib data dt
    statm = PATH.read_text()
    fields = statm.split()
    return int(fields[1]) * PAGESIZE


data = []
start_memory = get_resident_set_size()
for _ in range(10):
    data.append('X' * 100000)
    print(get_resident_set_size() - start_memory)

That produces a list that looks something like this:

0
0
368640
368640
368640
638976
638976
909312
909312
909312

You can see that it jumps by about 300,000 bytes after roughly 3 allocations of 100,000 bytes.

Answer 2

For Unix based systems (Linux, Mac OS X, Solaris), you can use the getrusage() function from the standard library module resource. The resulting object has the attribute ru_maxrss, which gives the peak memory usage for the calling process:

>>> resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
2656  # peak memory usage (kilobytes on Linux, bytes on OS X)

The Python docs don't make note of the units. Refer to your specific system's man getrusage.2 page to check the unit for the value. On Ubuntu 18.04, the unit is noted as kilobytes. On Mac OS X, it's bytes.

The getrusage() function can also be given resource.RUSAGE_CHILDREN to get the usage for child processes, and (on some systems) resource.RUSAGE_BOTH for total (self and child) process usage.

If you care only about Linux, you can alternatively read the /proc/self/status or /proc/self/statm file as described in other answers for this question and this one too.

Answer 3

Below is my function decorator which allows to track how much memory this process consumed before the function call, how much memory it uses after the function call, and how long the function is executed.

import time
import os
import psutil


def elapsed_since(start):
    return time.strftime("%H:%M:%S", time.gmtime(time.time() - start))


def get_process_memory():
    process = psutil.Process(os.getpid())
    return process.memory_info().rss


def track(func):
    def wrapper(*args, **kwargs):
        mem_before = get_process_memory()
        start = time.time()
        result = func(*args, **kwargs)
        elapsed_time = elapsed_since(start)
        mem_after = get_process_memory()
        print("{}: memory before: {:,}, after: {:,}, consumed: {:,}; exec time: {}".format(
            func.__name__,
            mem_before, mem_after, mem_after - mem_before,
            elapsed_time))
        return result
    return wrapper

So, when you have some function decorated with it

from utils import track

@track
def list_create(n):
    print("inside list create")
    return [1] * n

You will be able to see this output:

inside list create
list_create: memory before: 45,928,448, after: 46,211,072, consumed: 282,624; exec time: 00:00:00

Answer 4

import os, win32api, win32con, win32process
han = win32api.OpenProcess(win32con.PROCESS_QUERY_INFORMATION|win32con.PROCESS_VM_READ, 0, os.getpid())
process_memory = int(win32process.GetProcessMemoryInfo(han)['WorkingSetSize'])

Answer 5

On Windows, you can use WMI (home page, cheeseshop):

def memory():
    import os
    from wmi import WMI
    w = WMI('.')
    result = w.query("SELECT WorkingSet FROM Win32_PerfRawData_PerfProc_Process WHERE IDProcess=%d" % os.getpid())
    return int(result[0].WorkingSet)

On Linux (from python cookbook http://code.activestate.com/recipes/286222/:

import os
_proc_status = '/proc/%d/status' % os.getpid()

_scale = {'kB': 1024.0, 'mB': 1024.0*1024.0,
          'KB': 1024.0, 'MB': 1024.0*1024.0}

def _VmB(VmKey):
    '''Private.
    '''
    global _proc_status, _scale
     # get pseudo file  /proc/<pid>/status
    try:
        t = open(_proc_status)
        v = t.read()
        t.close()
    except:
        return 0.0  # non-Linux?
     # get VmKey line e.g. 'VmRSS:  9999  kB\n ...'
    i = v.index(VmKey)
    v = v[i:].split(None, 3)  # whitespace
    if len(v) < 3:
        return 0.0  # invalid format?
     # convert Vm value to bytes
    return float(v[1]) * _scale[v[2]]


def memory(since=0.0):
    '''Return memory usage in bytes.
    '''
    return _VmB('VmSize:') - since


def resident(since=0.0):
    '''Return resident memory usage in bytes.
    '''
    return _VmB('VmRSS:') - since


def stacksize(since=0.0):
    '''Return stack size in bytes.
    '''
    return _VmB('VmStk:') - since

Answer 6

Here is a useful solution that works for various operating systems, including Linux, Windows, etc.:

import os
import psutil
process = psutil.Process(os.getpid())
print(process.memory_info().rss)  # in bytes 

With Python 2.7 and psutil 5.6.3, the last line should be

print(process.memory_info()[0])

instead (there was a change in the API later).

Note: do pip install psutil if it is not installed yet.