objgraph

Memory leak using pandas dataframe

感情迁移 提交于 2019-12-17 16:35:32
问题 I am using pandas.DataFrame in a multi-threaded code (actually a custom subclass of DataFrame called Sound ). I have noticed that I have a memory leak, since the memory usage of my program augments gradually over 10mn, to finally reach ~100% of my computer memory and crash. I used objgraph to try tracking this leak, and found out that the count of instances of MyDataFrame is going up all the time while it shouldn't : every thread in its run method creates an instance, makes some calculations,

Why can't objgraph capture the growth of np.array()?

柔情痞子 提交于 2019-12-12 14:48:15
问题 See the code: import objgraph import numpy as np objgraph.show_growth() j = 20 y = [] for i in range(5): for l in range(j): y.append(np.array([np.random.randint(500),np.random.randint(500)])) print 'i:',i objgraph.show_growth() print '___' #objgraph.show_most_common_types(limit=100) j += 1 the result is: i: 1 wrapper_descriptor 1596 +3 weakref 625 +1 dict 870 +1 method_descriptor 824 +1 i: 2 i: 3 i: 4 For the 2,3 and 4 epoch, it shows nothing growing. But it should show that the number of

Memory leak using pandas dataframe

孤街醉人 提交于 2019-11-28 00:00:10
I am using pandas.DataFrame in a multi-threaded code (actually a custom subclass of DataFrame called Sound ). I have noticed that I have a memory leak, since the memory usage of my program augments gradually over 10mn, to finally reach ~100% of my computer memory and crash. I used objgraph to try tracking this leak, and found out that the count of instances of MyDataFrame is going up all the time while it shouldn't : every thread in its run method creates an instance, makes some calculations, saves the result in a file and exits ... so no references should be kept. Using objgraph I found that

Does Python GC deal with reference-cycles like this?

孤人 提交于 2019-11-27 07:40:53
Using objgraph , I found a bunch of objects like this: Will Python's garbage collector deal with cycles like this, or will it leak? A slightly wider view of the loop: Frédéric Hamidi Python's standard reference counting mechanism cannot free cycles, so the structure in your example would leak. The supplemental garbage collection facility , however, is enabled by default and should be able to free that structure, if none of its components are reachable from the outside anymore and they do not have __del__() methods . If they do, the garbage collector will not free them because it cannot

Does Python GC deal with reference-cycles like this?

≡放荡痞女 提交于 2019-11-26 13:46:06
问题 Using objgraph, I found a bunch of objects like this: Will Python's garbage collector deal with cycles like this, or will it leak? A slightly wider view of the loop: 回答1: Python's standard reference counting mechanism cannot free cycles, so the structure in your example would leak. The supplemental garbage collection facility, however, is enabled by default and should be able to free that structure, if none of its components are reachable from the outside anymore and they do not have __del__(