pytables

When running pd.read_hdf('myfile.h5') I get the following traceback error: [[...some longer traceback]] ~/.local/lib/python3.6/site-packages/pandas/io/pytables.py in read_array(self, key, start, stop) 2487 2488 if isinstance(node, tables.VLArray): -> 2489 ret = node[0][start:stop] 2490 else: 2491 dtype = getattr(attrs, 'value_type', None) ~/.local/lib/python3.6/site-packages/tables/vlarray.py in getitem (self, key) ~/.local/lib/python3.6/site-packages/tables/vlarray.py in read(self, start, stop, step) tables/hdf5extension.pyx in tables.hdf5extension.VLArray._read_array() ValueError: cannot set

I'm trying to multiplicate 2 big matrices with memory limit using hdf5 (pytables) but function numpy.dot seems to give me error: Valueerror: array is too big I need to do matrix multiplication by myself maybe blockwise or there is some another python function similar to numpy.dot? import numpy as np import time import tables import cProfile import numexpr as ne n_row=10000 n_col=100 n_batch=10 rows = n_row cols = n_col batches = n_batch atom = tables.UInt8Atom() #? filters = tables.Filters(complevel=9, complib='blosc') # tune parameters fileName_a = 'C:\carray_a.h5' shape_a = (rows*batches,

How can I create a huge numpy array using pytables. I tried this but gives me the "ValueError: array is too big." error: import numpy as np import tables as tb ndim = 60000 h5file = tb.openFile('test.h5', mode='w', title="Test Array") root = h5file.root h5file.createArray(root, "test", np.zeros((ndim,ndim), dtype=float)) h5file.close() b1r3k You could try to use tables.CArray class as it supports compression but... I think questions is more about numpy than pytables because you are creating array using numpy before storing it with pytables. In that way you need a lot of ram to execute np.zeros

I've been using pandas for research now for about two months to great effect. With large numbers of medium-sized trace event datasets, pandas + PyTables (the HDF5 interface) does a tremendous job of allowing me to process heterogenous data using all the Python tools I know and love. Generally speaking, I use the Fixed (formerly "Storer") format in PyTables, as my workflow is write-once, read-many, and many of my datasets are sized such that I can load 50-100 of them into memory at a time with no serious disadvantages. (NB: I do much of my work on Opteron server-class machines with 128GB+

I am reading data from a .mat file using the Pytables module. After reading the data, I want to insert this data into the database using psycopg. Here is a sample code piece: file = tables.openFile(matFile) x = 0 #populate the matData list for var in dest: data = file.getNode('/' + var)[:] matData.append(data) x = x+1 #insert into db for i in range(0,x): cur.execute("""INSERT INTO \"%s\" (%s) VALUES (%s)""" % tableName,dest[i],matData[i]) ) I am getting the following error: Traceback (most recent call last): File "./loadDBFromMAT.py", line 111, in <module> readInputFileAndLoad(args.matFileName