h5py

问题 I am trying to run a PyTorch implementation of a code, which is supposed to work on SBD dataset. The training labels are originally available in .bin file, which are then converted to HDF5 (.h5) files. Upon running the algorithm, I get an error as: " TypeError: h5py objects cannot be pickled " I think the error is stemming from torch.utils.data.DataLoader. Any idea if I am missing any concept here? I read that pickling is generally not preferred but as of now, my dataset is in HDF5 format

问题 I am trying to run a PyTorch implementation of a code, which is supposed to work on SBD dataset. The training labels are originally available in .bin file, which are then converted to HDF5 (.h5) files. Upon running the algorithm, I get an error as: " TypeError: h5py objects cannot be pickled " I think the error is stemming from torch.utils.data.DataLoader. Any idea if I am missing any concept here? I read that pickling is generally not preferred but as of now, my dataset is in HDF5 format

问题 I am trying to create hdf5 file, but the output file is empty. I have written a python code which is supposed to run in loop and write string in the created datasets. After the file gets saved, I found that the output file is always empty. Below is the piece of code I have written: h5_file_name = 'sample.h5' hf = h5py.File(h5_file_name, 'w') g1 = hf.create_group('Objects') dt = h5py.special_dtype(vlen=str) d1 = g1.create_dataset('D1', (2, 10), dtype=dt) d2 = g1.create_dataset('D2', (3, 10),

问题 I am trying to create hdf5 file, but the output file is empty. I have written a python code which is supposed to run in loop and write string in the created datasets. After the file gets saved, I found that the output file is always empty. Below is the piece of code I have written: h5_file_name = 'sample.h5' hf = h5py.File(h5_file_name, 'w') g1 = hf.create_group('Objects') dt = h5py.special_dtype(vlen=str) d1 = g1.create_dataset('D1', (2, 10), dtype=dt) d2 = g1.create_dataset('D2', (3, 10),

问题 creating a dataset from multiple hdf5 groups Code for groups with np.array(hdf.get('all my groups')) I have then added code for creating a dataset from groups. with h5py.File('/train.h5', 'w') as hdf: hdf.create_dataset('train', data=one_T+two_T+three_T+four_T+five_T) The error message being ValueError: operands could not be broadcast together with shapes(534456,4) (534456,14) The numbers in each group are the same other than the varying column lengths. 5 separate groups to one dataset. 回答1:

问题 I know that in c we can construct a compound dataset easily using struct type and assign data chunk by chunk. I am currently implementing a similar structure in Python with h5py . import h5py import numpy as np # we create a h5 file f = h5py.File("test.h5") # default is mode "a" # We define a compound datatype using np.dtype dt_type = np.dtype({"names":["image","feature"], "formats":[('<f4',(4,4)),('<f4',(10,))]}) # we define our dataset with 5 instances a = f.create_dataset("test", shape=(5,

问题 I know that in c we can construct a compound dataset easily using struct type and assign data chunk by chunk. I am currently implementing a similar structure in Python with h5py . import h5py import numpy as np # we create a h5 file f = h5py.File("test.h5") # default is mode "a" # We define a compound datatype using np.dtype dt_type = np.dtype({"names":["image","feature"], "formats":[('<f4',(4,4)),('<f4',(10,))]}) # we define our dataset with 5 instances a = f.create_dataset("test", shape=(5,

问题 I have a few questions wrapped up into this issue. I realize this might be a convoluted post and can provide extra details. A code package I use can produce large .h5 files (source.h5) (100+ Gb), where almost all of this data resides in 1 dataset (group2/D). I want to make a new .h5 file (dest.h5) using Python that contains all datasets except group2/D of source.h5 without needing to copy the entire file. I then will condense group2/D after some postprocessing and write a new group2/D in dest

问题 This is a follow up to this stackoverflow question Column missing when trying to open hdf created by pandas in h5py Where I am trying to create save a large amount of data onto a disk (too large to fit into memory), and retrieve sepecific rows of the data using indices. One of the solutions given in the linked post is to create a seperate key for every every row. At the moment I can only think of iterating through each row, and setting the keys directly. For example, if this is my data

问题 I'm currently working on a project regarding compression of HDF5 datasets and recently began using h5py. I followed the basic tutorials and was able to open,create and compress a file while it was being created. However, I've been unsuccessful when it comes to compressing an existing file (which is the aim of my work). I've tried opening files using 'r+' and then compressing chunked datasets but the file sizes have remained the same. Any suggestions on what commands to use or am I going about