How to implement a memory intensive python script for test

问题

I've applied a cgroups rule to a specific user, and I'd like to test whether memory of the programs running from the above user has been limited as expected. I tried with the following script:

import string
import random

if __name__ == '__main__':
    d = {}
    i = 0;
    for i in range(0, 100000000):
        val = ''.join(random.choice(string.ascii_uppercase + string.digits) for _ in range(200)) # generate ramdom string of size 200
        d[i] = val
        if i % 10000 == 0:
            print i

When I monitored the process via ps command, it turned out to be that the %MEM is increased to 4.8 and never changed when both cgroups service is on and off:

$ ps aux | grep mem_intensive.py
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
jason    11531 88.5  4.8 3312972 3191236 pts/0 R+   22:16   0:07 python mem_intensive.py

In this scenario, total memory is 62GB, thus 4.8% of it about 3GB. I set the limit to be 4GB without any other processes running on this user.

So could anyone give me some idea about this problematic python script? Thanks in advance.

回答1:

If you want to see whether cgroup works, just set the limit to 100MB and try to start the script. The point isn't to see whether a large limit works better or worse than a small one - you just want to make sure that a limit is enforced. For that, a small limit is enough.

To make sure that the dict grows as expected, you can print it's size using the answers to this question: Memory-usage of dictionary in Python?

回答2:

I've played a bit with your script, and it keeps growing, albeit slowly. The bottleneck is using random.choice. If you want to fill memory fast, generating randomness works against you. So just using fixed strings does exhaust the memory rather quickly. If using the following, while wanting to watch how it grows you'd probably throw a time.sleep() after your print:

if __name__ == '__main__':
    d = {}
    i = 0;
    for i in range(0, 100000000):
        d[i] = 'A'*1024
        if i % 10000 == 0:
            print(i)

filling memory faster:

just an one-liner:

['A'*1024 for _ in xrange(0, 1024*1024*1024)]

回答3:

range constructs a list in memory which your loop then iterates through, xrange creates a generator which is an object which feeds the loop like a sequence but does not build that sequence in memory. There is little difference between range and xrange for short ranges but significant differences for large ranges, quoting the Python docs: https://docs.python.org/2/library/functions.html#xrange

In Python 3, the functionality provided by xrange becomes the default for the range built-in. As a result of this, and the inherent memory advantage of xrange in Python 2, I've seen Python 2 to 3 compatibility layers map the Python 2 range function to call xrange instead, under the hood.

来源：https://stackoverflow.com/questions/30509322/how-to-implement-a-memory-intensive-python-script-for-test

标签

python

memory

cgroups