combining huge h5 files with multiple datasets into one with odo

问题

I have a a number of large (13GB+ in size) h5 files, each h5 file has two datasets that were created with pandas:

df.to_hdf('name_of_file_to_save', 'key_1',table=True) 
df.to_hdf('name_of_file_to_save', 'key_2', table=True) # saved to the same h5 file as above

I've seen a post here:

Concatenate two big pandas.HDFStore HDF5 files

on using odo to concatenate h5 files. What I want to do is for each h5 file that was created, each having key_1 and key_2, combine them so that all of the key_1 data are in one dataset in the new h5 file and all of the key_2 are in another dataset in the same new h5 file. All of key_1 have the same number of columns, the same applies to key_2

I've tried this:

from odo import odo
files = ['file1.h5','file2.h5','file3.h5','file4.h5']
for i in files:
    odo('hdfstore://path_to_here_h5_files_live/%s::key_1' % i,
        'hdfstore://path_store_new_large_h5::key_1')

Howeever I get an error:

(tables/hdf5extension.c:7824)
tables.exceptions.HDF5ExtError: HDF5 error back trace

File "H5A.c", line 259, in H5Acreate2
  unable to create attribute
File "H5Aint.c", line 275, in H5A_create
  unable to create attribute in object header
File "H5Oattribute.c", line 347, in H5O_attr_create
  unable to create new attribute in header
File "H5Omessage.c", line 224, in H5O_msg_append_real
  unable to create new message
File "H5Omessage.c", line 1945, in H5O_msg_alloc
  unable to allocate space for message
File "H5Oalloc.c", line 1142, in H5O_alloc
  object header message is too large

End of HDF5 error back trace

Can't set attribute 'non_index_axes' in node:
/key_1 (Group) ''.
Closing remaining open 

回答1:

For this specific case it was a matter of having too many columns, which exceeded the memory limit allocated for that piece of information. The solution is to load the dataframe and transpose it.

来源：https://stackoverflow.com/questions/38283279/combining-huge-h5-files-with-multiple-datasets-into-one-with-odo