How to restore a model by filename in Tensorflow r12?

Question

I have run the distributed mnist example: https://github.com/tensorflow/tensorflow/blob/r0.12/tensorflow/tools/dist_test/python/mnist_replica.py

Though I have set th

Answer 1

OK, I can answer my own question. What I found was that my python script was adding an extra '/' to my path so I was executing: saver.restore(sess,'/path/to/train//model.ckpt-1234')

somehow that was causing a problem with tensorflow.

When I removed it, calling: saver.restore(sess,'/path/to/trian/model.ckpt-1234')

it worked as expected.

Answer 2

I also used Tensorlfow r0.12 and I didn't think there is any issue for saving and restoring model. The following is a simple code that you can have a try:

import tensorflow as tf

# Create some variables.
v1 = tf.Variable(tf.random_normal([784, 200], stddev=0.35), name="v1")
v2 = tf.Variable(tf.random_normal([784, 200], stddev=0.35), name="v2")

# Add an op to initialize the variables.
init_op = tf.global_variables_initializer()

# Add ops to save and restore all the variables.
saver = tf.train.Saver()

# Later, launch the model, initialize the variables, do some work, save the
# variables to disk.
with tf.Session() as sess:
  sess.run(init_op)
  # Do some work with the model.

  # Save the variables to disk.
  save_path = saver.save(sess, "/tmp/model.ckpt")
  print("Model saved in file: %s" % save_path)

# Later, launch the model, use the saver to restore variables from disk, and
# do some work with the model.
with tf.Session() as sess:
  # Restore variables from disk.
  saver.restore(sess, "/tmp/model.ckpt")
  print("Model restored.")
  # Do some work with the model

although in r0.12, the checkpoint is stored in multiple files, you can restore it by using the common prefix, which is 'model.ckpt' in your case.

Answer 3

You can restore the model like this:

saver = tf.train.import_meta_graph('./src/models/20170512-110547/model-20170512-110547.meta')
            saver.restore(sess,'./src/models/20170512-110547/model-20170512-110547.ckpt-250000'))

Where the path '/src/models/20170512-110547/' contains three files:

model-20170512-110547.meta
model-20170512-110547.ckpt-250000.index
model-20170512-110547.ckpt-250000.data-00000-of-00001

And if in one directory there are more than one checkpoints,eg: there are checkpoint files in the path ./20170807-231648/:

checkpoint     
model-20170807-231648-0.data-00000-of-00001   
model-20170807-231648-0.index    
model-20170807-231648-0.meta   
model-20170807-231648-100000.data-00000-of-00001   
model-20170807-231648-100000.index   
model-20170807-231648-100000.meta

you can see that there are two checkpoints, so you can use this:

saver =    tf.train.import_meta_graph('/home/tools/Tools/raoqiang/facenet/models/facenet/20170807-231648/model-20170807-231648-0.meta')

saver.restore(sess,tf.train.latest_checkpoint('/home/tools/Tools/raoqiang/facenet/models/facenet/20170807-231648/'))

Answer 4

use only model.ckpt-1234

at least it works for me

Answer 5

I'm new to TF and met the same issue. After reading Yuan Ma's comments, I copied the '.index' to the same 'train\ckpt' folder together with '.data-00000-of-00001' file. Then it worked! So, the .index file is sufficient when restoring the models. I used TF on Win7, r12.

Answer 6

The R12 has changed the checkpoint format. You should save the model in the old format.

import tensorflow as tf
from tensorflow.core.protobuf import saver_pb2
...
saver = tf.train.Saver(write_version = saver_pb2.SaverDef.V1)
saver.save(sess, './model.ckpt', global_step = step)

According to the TensorFlow v0.12.0 RC0’s release note:

New checkpoint format becomes the default in tf.train.Saver. Old V1 checkpoints continue to be readable; controlled by the write_version argument, tf.train.Saver now by default writes out in the new V2 format. It significantly reduces the peak memory required and latency incurred during restore.

see details in my blog.