Is Session.run(fetches) guaranteed to execute its “fetches” arguments in-order?

Question

Is Session.run(fetches, feed_dict) guaranteed to execute its fetches arguments in-order?

The documentation doesn\'t seem to mention it.

Answer 1

No. By default, Tensorflow is free to evaluate operators in any order. Because of concurrency, that order may even change between runs. This is usually a good thing because it means that Tensorflow may make optimal use of the available hardware. It can be problematic if your code mutates state such as Variables.

However, if for some reason you do wish to control the order of evaluation, in general you can use control dependencies to enforce an order between operators. Control dependencies are documented here:

https://www.tensorflow.org/api_docs/python/tf/Graph#control_dependencies

Hope that helps!

Answer 2

After posting this, and during the discussion in

Is it possible to get the objective function value during each training step?

I noticed that the execution order is undefined. For example, consider this code:

import tensorflow as tf
x = tf.Variable(0, dtype=tf.float32)
loss = tf.nn.l2_loss(x-1)
train_opt = tf.train.GradientDescentOptimizer(1)
train_op = train_opt.minimize(loss)
init_op = tf.global_variables_initializer()

with tf.Session() as sess:
    sess.run(init_op)
    print sess.run([x, train_op])

With TensorFlow 1.1, if the environment variable CUDA_VISIBLE_DEVICES is set to one of the GPUs, this prints

[0.0, None]

and if it is set to "", this code prints

[1.0, None]

Unfortunately, I don't see anything in the documentation specifying the execution order or warning users that it's undefined.