At what stage is a tensorflow graph set up?

Question

An optimizer typically run the same computation graph for many steps until convergence. Does tensorflow setup the graph at the beginning and reuse it for every step? What if

Answer 1

TensorFlow exposes only one graph that is visible to the user, namely the one specified by the user. The user can run the graph with Session.run() or by calling Tensor.eval() on some tensor. A Session.run() call can specify some tensors to be fed and others to be fetched. Depending on what needs to be fetched, the TensorFlow runtime could be internally constructing and optimizing various data structures, including a pruned version of the user visible graph. However, this internal graph is not visible to the user in anyway. No, TensorFlow doesn't 'pre-generate' all possible graphs. Yes, TensorFlow does perform extensive optimizations on the computation graph. And finally, changing the batch size of a tensor that is fed doesn't change the structure of the graph.