Import multiple csv files into pandas and concatenate into one DataFrame

前端 未结 16 1771
既然无缘
既然无缘 2020-11-21 07:47

I would like to read several csv files from a directory into pandas and concatenate them into one big DataFrame. I have not been able to figure it out though. Here is what I

16条回答
  •  花落未央
    2020-11-21 08:11

    Alternative using the pathlib library (often preferred over os.path).

    This method avoids iterative use of pandas concat()/apped().

    From the pandas documentation:
    It is worth noting that concat() (and therefore append()) makes a full copy of the data, and that constantly reusing this function can create a significant performance hit. If you need to use the operation over several datasets, use a list comprehension.

    import pandas as pd
    from pathlib import Path
    
    dir = Path("../relevant_directory")
    
    df = (pd.read_csv(f) for f in dir.glob("*.csv"))
    df = pd.concat(df)
    

提交回复
热议问题