How to use the Tensorflow Dataset Pipeline for Variable Length Inputs?

Question

I am training a Recurrent Neural Network in Tensorflow over a dataset of sequence of numbers of varying lengths and have been trying to use the tf.data API to creat

Answer 1

As you have noticed, tf.data.Dataset.from_tensor_slices() only works on objects that can be converted to a (dense) tf.Tensor or a tf.SparseTensor. The easiest way to get variable-length NumPy data into a Dataset is to use tf.data.Dataset.from_generator(), as follows:

dataset = tf.data.Dataset.from_generator(lambda: dataset_list, 
                                         tf.as_dtype(dataset_list[0].dtype),
                                         tf.TensorShape([None, 32, 2]))