Loading big CSV file with pandas

前端 未结 2 1851
独厮守ぢ
独厮守ぢ 2021-02-10 17:53

I am trying to load a csv file (around 250 MB) as dataframe with pandas. In my first try I used the typical read_csv command but I receive an Error memory. I have tried the appr

相关标签:
2条回答
  • 2021-02-10 18:11

    I suggest that you install the 64 Bit version of winpython. Then you should be able to load a 250 MB file without problems.

    0 讨论(0)
  • 2021-02-10 18:11

    I'm late, but the actual problem with the posted code is that using pd.concat([chunk for chunk in x]) effectively cancels any benefit of chunking because it concatenates all those chunks into one big DataFrame again.
    That probably even requires twice the memory temporarily.

    0 讨论(0)
提交回复
热议问题