问题
I have a set of python processes that share a large object (sharing is done by forking the processes after initializing the object)
I notice a strange memory leak:
- the process memory (VSZ and RSS) hardly change
- the total system memory increases steadily
My guess is the shared object does change (it's 'logically' readonly, but it's possible that some internal private variables change even when just reading from it) - which causes memory pages to be copied
is there a way to verify this?
回答1:
To answer your specific question "is there a way to verify this?", if I understand it correctly, you can do the following if you want to see whether there are any changes associated with the pages that contain the large object.
1) Determine the address of your "large shared object" and the address of where that object ends.
2) If the start address is not on a 4K page boundary, round the start address down to the page boundary before where the object starts.
3) If the end address is not on a 4K boundary, round the end address up to the page boundary after where the object ends.
4) Dump that memory range for the process and all its children to separate files and compare them.
However, I think that Will os.fork() use copy on write or do a full copy of the parent-process in Python? may already provide an explanation for at least some of the writing that is requiring copying. Specifically, python objects are reference counted and your child processes will be altering reference counts.
Have you considered using python's threading rather than creating child processes?
回答2:
You can give the following a try:
dtrace -n wp_page_copy:entry'/pid == 11111/ { @counts[ustack()] = count(); }'
Not sure if wp_page_copy is restricted to only being used during copy-on-write. I seem to recall reading that it is, but I wouldn't bet on it without confirming. You'll want to change 11111 to your python pid.
来源：https://stackoverflow.com/questions/51036943/is-it-possible-to-monitor-copy-on-write-for-forked-linux-processes-specificall