MemoryError when I merge two Pandas data frames

Question

I searched almost all over the internet and somehow none of the approaches seem to work in my case.

I have two large csv files (each with a million+ rows and about

Answer 1

In general chunk version suggested by @T_cat works great.

However, memory exploding might be caused by joining on columns that have Nan values. So you may want to exclude those rows from the join.

see: https://github.com/pandas-dev/pandas/issues/24698#issuecomment-614347153

Answer 2

The reason you might be getting MemoryError: Unable to allocate.. could be due to duplicates or blanks in your dataframe. Check the column you are joining on (when using merge) and see if you have duplicates or blanks. If so get rid of them using this command:

df.drop_duplicates(subset ='column_name', keep = False, inplace = True) 

Then re-run your python/pandas code. This worked for me.

Answer 3

When you are merging data using pandas.merge it will use df1 memory, df2 memory and merge_df memory. I believe that it is why you get a memory error. You should export df2 to a csv file and use chunksize option and merge data.

It might be a better way but you can try this. *for large data set you can use chunksize option in pandas.read_csv

df1 = pd.read_csv("yourdata.csv")
df2 = pd.read_csv("yourdata2.csv")
df2_key = df2.Colname2

# creating a empty bucket to save result
df_result = pd.DataFrame(columns=(df1.columns.append(df2.columns)).unique())
df_result.to_csv("df3.csv",index_label=False)

# save data which only appear in df1 # sorry I was doing left join here. no need to run below two line.
# df_result = df1[df1.Colname1.isin(df2.Colname2)!=True]
# df_result.to_csv("df3.csv",index_label=False, mode="a")

# deleting df2 to save memory
del(df2)

def preprocess(x):
    df2=pd.merge(df1,x, left_on = "Colname1", right_on = "Colname2")
    df2.to_csv("df3.csv",mode="a",header=False,index=False)

reader = pd.read_csv("yourdata2.csv", chunksize=1000) # chunksize depends with you colsize

[preprocess(r) for r in reader]

this will save merged data as df3.