Recently I have been trying to learn to use TensorFlow, and I do not understand how variable scopes work exactly. In particular, I have the following problem:
The answer above is somehow misguiding.
Let me answer why you got two different scope names, even though it looks like that you defined two identical functions: creating
and creating_mod
.
This is simply because you used tf.Variable(0.0, name=s)
to create the variable in the function creating_mod
.
ALWAYS use tf.get_variable
, if you want your variable to be recognized by scope!
Check out this issue for more details.
Thanks!
The "_2"
in "BasicLSTMCell_2"
relates to the name scope in which the op outpts[2]
was created. Every time you create a new name scope (with tf.name_scope()) or variable scope (with tf.variable_scope()) a unique suffix is added to the current name scope, based on the given string, possibly with an additional suffix to make it unique. The call to rnn.rnn(...)
has the following pseudocode (simplified and using public API methods for clarity):
outputs = []
with tf.variable_scope("RNN"):
for timestep, input_t in enumerate(inputs):
if timestep > 0:
tf.get_variable_scope().reuse_variables()
with tf.variable_scope("BasicLSTMCell"):
outputs.append(...)
return outputs
If you look at the names of the tensors in outpts
, you'll see that they are the following:
>>> print [o.name for o in outpts]
[u'RNN/BasicLSTMCell/mul_2:0',
u'RNN/BasicLSTMCell_1/mul_2:0',
u'RNN/BasicLSTMCell_2/mul_2:0',
u'RNN/BasicLSTMCell_3/mul_2:0',
u'RNN/BasicLSTMCell_4/mul_2:0']
When you enter a new name scope (by entering a with tf.name_scope("..."): or with tf.variable_scope("..."): block), TensorFlow creates a new, unique name for the scope. The first time the "BasicLSTMCell"
scope is entered, TensorFlow uses that name verbatim, because it hasn't been used before (in the "RNN/"
scope). The next time, TensorFlow appends "_1"
to the scope to make it unique, and so on up to "RNN/BasicLSTMCell_4"
.
The main difference between variable scopes and name scopes is that a variable scope also has a set of name-to-tf.Variable bindings. By calling tf.get_variable_scope().reuse_variables()
, we instruct TensorFlow to reuse rather than create variables for the "RNN/"
scope (and its children), after timestep 0. This ensures that the weights are correctly shared between the multiple RNN cells.