Why should I build separated graph for training and validation in tensorflow?

Question

I\'ve been using tensorflow for a while now. At first I had stuff like this:

def myModel(training):
    with tf.scope_variables(\'model\', reuse=not training         


        

Answer 1

tf.estimator.Estimator classes indeed create a new graph for each invocation and this has been the subject of furious debates, see this issue on GitHub. Their approach is to build the graph from scratch on each train, evaluate and predict invocations and restore the model from the last checkpoint. There are clear downsides of this approach, for example:

• A loop that calls train and evaluate will create two new graphs on every iteration.
• One can't evaluate while training easily (though there are workarounds, train_and_evaluate, but this doesn't look very nice).

I tend to agree that having the same graph and model for all actions is convenient and I usually go with this solution. But in a lot of cases when using a high-level API like tf.estimator.Estimator, you don't deal with the graph and variables directly, so you shouldn't care how exactly the model is organized.

Answer 2

You do not have to use two neural nets for training and validation. After all, as you noticed, tensorflow helps you having a monolothical train-and-validate net by allowing the training parameter of some layers to be a placeholder.

However, why wouldn't you? By having separate nets for training and for validation, you set yourself on the right path and future-proof your code. Your training and validation nets might be identical today, but you might later see some benefit to having distinct nets such as having different inputs, different outputs, removing out intermediate layers, etc.

Also, because variables are shared between them, having distinct training and validation nets comes at almost no penalty.

So, keeping a single net is fine; in my experience though, any project other than playful experimentation is likely to implement a distinct validation net at some point, and tensorflow makes it easy to do just that with minimal penalty.