pytables

Does PyTables support storing Python objects? something like this : dtype = np.dtype([('Name', '|S2'), ('objValue', object)]) data = np.zeros(3, dtype) file.createArray(box3,'complicated',data) I get error when trying to do this of course... How to properly store arrays of objects?Is it possible? Thanks Try the pickle module if you want to store complicated data somewhere it isn't supported by the library in question. You can save generic Python object with Pytables: >>> dtype = np.dtype([('Name', '|S2'), ('objValue', object)]) >>> data = np.zeros(3, dtype) >>> file = tables.openFile('/tmp

I'm trying to compute the difference in pixel values of two images, but I'm running into memory problems because the images I have are quite large. Is there way in python that I can read an image lets say in 10x10 chunks at a time rather than try to read in the whole image? I was hoping to solve the memory problem by reading an image in small chunks, assigning those chunks to numpy arrays and then saving those numpy arrays using pytables for further processing. Any advice would be greatly appreciated. Regards, Berk You can use numpy.memmap and let the operating system decide which parts of the

I have the following problem: I have a set several hdf5 files with similar data frames which I want to sort globally based on multiple columns. My input is the file names and an ordered list of columns I want to use for sorting. The output should be a single hdf5 file containing all the sorted data. Each file can contain millions of rows. I can afford loading a single file in memory but not the entire dataset. Naively I would like first to copy all the data in a single hdf5 file (which is not difficult) and then find out a way to do in memory sorting of this huge file. Is there a quick way to

I am creating a Python desktop application that allows users to select different distributional forms to model agricultural yield data. I have the time series agricultural data - close to a million rows - saved in a SQLite database (although this is not set in stone if someone knows of a better choice). Once the user selects some data, say corn yields from 1990-2010 in Illinois, I want them to select a distributional form from a drop-down. Next, my function fits the distribution to the data and outputs 10,000 points drawn from that fitted distributional form in a Numpy array. I would like this