Most efficient way to use a large data set for PyTorch?

Question

Perhaps this question has been asked before, but I\'m having trouble finding relevant info for my situation.

I\'m using PyTorch to create a CNN for regression with image

Answer 1

For speed I would advise to used HDF5 or LMDB:

Reasons to use LMDB:

LMDB uses memory-mapped files, giving much better I/O performance. Works well with really large datasets. The HDF5 files are always read entirely into memory, so you can’t have any HDF5 file exceed your memory capacity. You can easily split your data into several HDF5 files though (just put several paths to h5 files in your text file). Then again, compared to LMDB’s page caching the I/O performance won’t be nearly as good. [http://deepdish.io/2015/04/28/creating-lmdb-in-python/]

If you decide to used LMDB:

ml-pyxis is a tool for creating and reading deep learning datasets using LMDBs.

It allows to create binary blobs (LMDB) and they can be read quite fast. The link above comes with some simple examples on how to create and read the data. Including python generators/ iteratos .

This notebook has an example on how to create a dataset and read it paralley while using pytorch.

If you decide to use HDF5:

PyTables is a package for managing hierarchical datasets and designed to efficiently and easily cope with extremely large amounts of data.

https://www.pytables.org/