Tensorflow Dataset.from_generator fails with pyfunc exception

Question

I am trying tensorflow\'s nightly 1.4 as I need Dataset.from_generator to stich together some variable length datasets. This simple code (idea from here):

im         


        

Answer 1

The Dataset.from_generator() method is designed to connect non-TensorFlow Python code to a tf.data input pipeline. For example, you can yield simple Python objects (such as int and str objects), lists, or NumPy arrays from a generator, and they will be converted into TensorFlow values.

However, in your example code, you are yielding the result of it.get_next() which is a tf.Tensor object. This is not supported. If you need to capture an iterator in a different dataset, you can use Dataset.map() over a dummy dataset, as follows:

import tensorflow as tf

Dataset = tf.contrib.data.Dataset
it2 = Dataset.range(5).make_one_shot_iterator()

das_dataset = Dataset.from_tensors(0).repeat().map(lambda _: it2.get_next())
das_dataset_it = das_dataset.make_one_shot_iterator()
with tf.Session() as sess:
    while True:
        print(sess.run(it2.get_next()))
        print(sess.run(das_dataset_it.get_next()))