How to append multiple pandas DataFrames in a loop?

Question

I\'ve been banging my head on this python problem for a while and am stuck. I am for-looping through several csv files and want one data frame that appends the csv files in a wa

Answer 1

I'd recommend first concatenating all your dataframes together with pd.concat, and then doing one final pivot operation.

filenames = sorted(glob.glob('*.csv'))

new = [pd.read_csv(f, parse_dates=['time_stamp']) for f in filenames]
df = pd.concat(new) # omit axis argument since it is 0 by default

df = df.pivot(index='time_stamp', columns='pod')

Note that I'm forcing read_csv to parse time_stamp when loading the dataframe, so parsing after loading is no longer required.

---

MCVE

df

   pod  time_stamp      value
0   97  2016-02-22   3.048000
1   97  2016-02-29  23.622001
2   97  2016-03-07  13.970001
3   97  2016-03-14   6.604000
4   97  2016-03-21        NaN

df.pivot(index='time_stamp', columns='pod')

                value
pod                97
time_stamp           
2016-02-22   3.048000
2016-02-29  23.622001
2016-03-07  13.970001
2016-03-14   6.604000
2016-03-21        NaN