Using a variable for num_splits for tf.split()

Question

Is it possible to use a placeholder input for the num_split parameter to tf.split()?

I would ideally like to do something like:

num_splits = tf.place         


        

Answer 1

A solution to a similar problem in an unrelated circumstance is using a placeholder for a fixed length of maximum size. Suppose I have a sequence of length 40. t = tf.range(40). Now at run time, I get a split, (say x = [6,9,10,5,1]). Now follow the steps

Step 1: Determine the maximum number of splits there can be, say: 19

Step 2:

    num_splits = tf.placeholder(tf.int32, [19]) 
    y= tf.split(t, num_or_size_splits=num_splits, axis=0)

This will break the sequence into run-time determined split sizes

Step 4: At run time :

    x = [6,9,10,5,1] 
    x += [40-sum(x)] + [0 for i in range(19-1-len(x))]

First line means the actual split sizes we need
Split requires the split sizes should sum upto total split size, i.e. 40 in this case, and 0 are the split sizes for the left over splits.

session.run(y, feed_dict={num_splits:x}) will show the results like :

[0, 1, 2, 3, 4, 5]  
[ 6,  7,  8,  9, 10, 11, 12, 13, 14]  
[15, 16, 17, 18, 19, 20, 21, 22, 23, 24]  
[25, 26, 27, 28, 29]  
[30]  
[31, 32, 33, 34, 35, 36, 37, 38, 39]  
[]
.
.
[]  

Step 5: (Optional, preferred) pad with zeros till the max length of the sequence

    def pad_up_to(t, max_in_dims, constant_values):
        s = tf.shape(t)
        paddings = [[0, m-s[i]] for (i,m) in enumerate(max_in_dims)]
        return tf.pad(t, paddings, 'CONSTANT', constant_values=constant_values)


    m = []
    for t1 in y :
      t1=tf.reshape(t1,[1,-1])
      t_padded = pad_up_to(t1, [1,15], 0)
      session.run(t_padded  , feed_dict={num_splits:x})  
      m+=  [t_padded]
    m= tf.concat(m,0)

This will pad the chunks with zeros to create equal sized chunks.

NOTE: The above methodology helped me in converting sequences into sentences (variable number of sentences) for NLP related tasks

funciton : pad_up_to() is taken from Q:42334646

Answer 2

There's a general philosophy of "tensor in-tensor out" for core graph ops, so it may simplify things if you can restructure your computation to deal with single tensor of variable size instead of variable number of tensors.

Ops like pack, unpack, split deal with multiple tensors but they compile into "tensor-in/tensor-out" ops during graph construction time, which is why num_splits needs to be fixed. Ops like dynamic_partition, dynamic_stitch, dequeue_many take over some of that functionality for single tensors with variable 0-th dimension.

If you really need to deal with variable number of tensors, typical approach is break computation into multiple session.run calls, with one input tensor per run call, and tie things together using queues. There's a slice_input_producer which splits variable sized input along 0'th dimension and produces a tensor for each row, so if you wanted to evaluate myfunction in a loop on each row of inputs you could do this

def myfunction(vector):
  result = tf.reduce_sum(vector)
  print_result = tf.Print(result, [result], "myfunction called ")
  return print_result

MAX_ROWS = 10

# input matrix with 2 columns and unknown number of rows (<MAX_ROWS)
inputs = tf.placeholder(tf.int32, [None, 2])
# copy of inputs, will need to have a persistent copy of it because we will
# be fetching rows in different session.run calls
data = tf.Variable(inputs, validate_shape=False)
# input producer that iterates over the rows and pushes them onto Queue
row = tf.train.slice_input_producer([data], num_epochs=1, shuffle=False)[0]
myfunction_op = myfunction(row)

# this op will save placeholder values into the variable
init_op = tf.initialize_all_variables()

# Coordinator is not necessary in this case, but you'll need it if you have
# more than one Queue in order to close all queues together
sess = tf.Session()
coord = tf.train.Coordinator()
threads = tf.train.start_queue_runners(sess=sess, coord=coord)

sess.run([init_op], feed_dict={inputs:[[0, 0], [1, 1], [2, 2]]})

try:
  for i in range(MAX_ROWS):
    sess.run([myfunction_op])
except tf.errors.OutOfRangeError:
  print('Done iterating')
finally:
  # When done, ask other threads to stop.
  coord.request_stop()

If you run this, you should see

myfunction called [0]
myfunction called [2]
myfunction called [4]
Done iterating