How should I use the h5py library for storing time series data?

问题

I have some time series data that i previously stored as hdf5 files using pytables. I recently tried storing the same with h5py lib. However, since all elements of numpy array have to be of same dtype, I have to convert the date (which is usually the index) into 'float64' type before storing it using h5py lib. When I use pytables, the index and its dtype are preserved which makes it possible for me to query the time-series without the need of pulling it all in the memory. I guess with h5py that is not possible. am I missing something here? And if not, under what situations should i use h5py lib to store time series data? I ask this question cause, clarity on this could help me design a more efficient (processing & storage wise) project.

below is simple code, where I have to lose index information in order to store it as a single dtype object

dt_range = pd.date_range('2016-12-01','2016-12-10')
data = np.arange(0,20).reshape(-1,2)
df = pd.DataFrame(data,index = dt_range, columns = list('ab'), dtype = 'float')
df.index  = df.index.to_julian_date()
df = df.reset_index()
h = h5py.File(r'path\temp.h5', 'w')
dset = h.create_dataset('temp',data = df.values, shape = (10,3))

回答1:

I'd use pandas to_hdf

dt_range = pd.date_range('2016-12-01','2016-12-10')
data = np.arange(0,20).reshape(-1,2)
df = pd.DataFrame(data,index = dt_range, columns = list('ab'), dtype = 'float')
df.index  = df.index.to_julian_date()
df = df.reset_index()

with pd.HDFStore('temp.h5', 'w') as h:
    df.to_hdf(h, 'temp')

pd.read_hdf('temp.h5', 'temp')

回答2:

When I run @piRSquared code, and look at the file with h5py I see:

In [4]: import h5py
In [5]: f=h5py.File('temp.h5')

In [8]: list(f.keys())
Out[8]: ['temp']
In [9]: f['temp']
Out[9]: <HDF5 group "/temp" (4 members)>
In [10]: list(f['temp'].keys())
Out[10]: ['axis0', 'axis1', 'block0_items', 'block0_values']

In [11]: f['temp']['axis0'][:]
Out[11]: 
array([b'index', b'a', b'b'], 
      dtype='|S5')
In [12]: f['temp']['axis1'][:]
Out[12]: array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], dtype=int64)
In [13]: f['temp']['block0_items'][:]
Out[13]: 
array([b'index', b'a', b'b'], 
      dtype='|S5')
In [14]: f['temp']['block0_values'][:]
Out[14]: 
array([[  2.45772350e+06,   0.00000000e+00,   1.00000000e+00],
       [  2.45772450e+06,   2.00000000e+00,   3.00000000e+00],
       [  2.45772550e+06,   4.00000000e+00,   5.00000000e+00],
       [  2.45772650e+06,   6.00000000e+00,   7.00000000e+00],
       [  2.45772750e+06,   8.00000000e+00,   9.00000000e+00],
       [  2.45772850e+06,   1.00000000e+01,   1.10000000e+01],
       [  2.45772950e+06,   1.20000000e+01,   1.30000000e+01],
       [  2.45773050e+06,   1.40000000e+01,   1.50000000e+01],
       [  2.45773150e+06,   1.60000000e+01,   1.70000000e+01],
       [  2.45773250e+06,   1.80000000e+01,   1.90000000e+01]])

So it has saved the indexing information in 3 series, and the values in another, which loads as a 2d numpy array.

That's the same kind of information that I'd expect to see from a file created by pytables.

According to it's documentation, pd.HDFStore is using pytables.

来源：https://stackoverflow.com/questions/41173254/how-should-i-use-the-h5py-library-for-storing-time-series-data

标签

python