Import multiple csv files into pandas and concatenate into one DataFrame

前端 未结 16 1729
既然无缘
既然无缘 2020-11-21 07:47

I would like to read several csv files from a directory into pandas and concatenate them into one big DataFrame. I have not been able to figure it out though. Here is what I

16条回答
  •  一生所求
    2020-11-21 08:00

    If you want to search recursively (Python 3.5 or above), you can do the following:

    from glob import iglob
    import pandas as pd
    
    path = r'C:\user\your\path\**\*.csv'
    
    all_rec = iglob(path, recursive=True)     
    dataframes = (pd.read_csv(f) for f in all_rec)
    big_dataframe = pd.concat(dataframes, ignore_index=True)
    

    Note that the three last lines can be expressed in one single line:

    df = pd.concat((pd.read_csv(f) for f in iglob(path, recursive=True)), ignore_index=True)
    

    You can find the documentation of ** here. Also, I used iglobinstead of glob, as it returns an iterator instead of a list.



    EDIT: Multiplatform recursive function:

    You can wrap the above into a multiplatform function (Linux, Windows, Mac), so you can do:

    df = read_df_rec('C:\user\your\path', *.csv)
    

    Here is the function:

    from glob import iglob
    from os.path import join
    import pandas as pd
    
    def read_df_rec(path, fn_regex=r'*.csv'):
        return pd.concat((pd.read_csv(f) for f in iglob(
            join(path, '**', fn_regex), recursive=True)), ignore_index=True)
    

提交回复
热议问题