Python 2.7: Appending Data to Table in Pandas

问题

I am reading data from image files and I want to append this data into a single HDF file. Here is my code:

datafile = pd.HDFStore(os.path.join(path,'imageData.h5'))
for file in fileList: 
     data = {'X Position' :  pd.Series(xpos, index=index1),
             'Y Position' :  pd.Series(ypos, index=index1),
             'Major Axis Length' :  pd.Series(major, index=index1),
             'Minor Axis Length' :  pd.Series(minor, index=index1), 
             'X Velocity' :  pd.Series(xVelocity, index=index1),
             'Y Velocity' :  pd.Series(yVelocity, index=index1) }
    df = pd.DataFrame(data)
    datafile['df'] = df
    datafile.close()

This is obviously incorrect as it overwrites each set of data with the new one each time the loop runs.

If instead of datafile['df'] = df, I use

datafile.append('df',df)

df.to_hdf(os.path.join(path,'imageData.h5'), 'df', append=True, format = 'table')

I get the error:

ValueError: Can only append to Tables

I have referred to the documentation and other SO questions, without avail.

So, I am hoping someone can explain why this isn't working and how I can successfully append all the data to one file. I am willing to use a different method (perhaps pyTables) if necessary.

Any help would be greatly appreciated.

回答1:

This will work in 0.11. Once you create a group (e.g the label where you are storing data, the 'df' here). If you store a fixed format it will overwrite (and if you try to append will give you the above error msg); if you write a table format you can append. Note that in 0.11, to_hdf does not correctly pass keywords thru to the underlying function so you can use it ONLY to write a fixed format.

datafile = pd.HDFStore(os.path.join(path,'imageData.h5'),mode='w')
for file in fileList: 
     data = {'X Position' :  pd.Series(xpos, index=index1),
             'Y Position' :  pd.Series(ypos, index=index1),
             'Major Axis Length' :  pd.Series(major, index=index1),
             'Minor Axis Length' :  pd.Series(minor, index=index1), 
             'X Velocity' :  pd.Series(xVelocity, index=index1),
             'Y Velocity' :  pd.Series(yVelocity, index=index1) }
    df = pd.DataFrame(data)
    datafile.append('df',df)
datafile.close

来源：https://stackoverflow.com/questions/22035360/python-2-7-appending-data-to-table-in-pandas

标签

python

pandas

hdf5

pytables