tensorflow-estimator | 易学教程

Getting free text features into Tensorflow Canned Estimators with Dataset API via feature_columns

阅读更多关于 Getting free text features into Tensorflow Canned Estimators with Dataset API via feature_columns

问题 I'm trying to build a model that gives reddit_score = f('subreddit','comment') Mainly this is as an example i can then build on for a work project. My code is here. My problem is that i see that canned estimators e.g. DNNLinearCombinedRegressor must have feature_columns that are part of FeatureColumn class. I have my vocab file and know that if i was to just limit to the first word of a comment i could just do something like tf.feature_column.categorical_column_with_vocabulary_file( key=

How to use tf.data's initializable iterator and reinitializable interator and feed data to estimator api?

阅读更多关于 How to use tf.data's initializable iterator and reinitializable interator and feed data to estimator api?

问题 All the official google tutorials use the one shot iterator for all the estimator api implementation, i couldnt find any documentation on how to use tf.data's initializable iterator and reinitializable interator instead of one shot iterator. Can someone kindly show me how to switch between train_data and test_data using tf.data's initializable iterator and reinitializable interator. We need to run a session to use feed dict and switch the dataset in the initializable iterator, its a low level

How to get the last global_step from an tf.estimator.Estimator

阅读更多关于 How to get the last global_step from an tf.estimator.Estimator

How can I obtain the last global_step from a tf.estimator.Estimator after train(...) finishes? For instance, a typical Estimator-based training routine might be set up like this: n_epochs = 10 model_dir = '/path/to/model_dir' def model_fn(features, labels, mode, params): # some code to build the model pass def input_fn(): ds = tf.data.Dataset() # obviously with specifying a data source # manipulate the dataset return ds run_config = tf.estimator.RunConfig(model_dir=model_dir) estimator = tf.estimator.Estimator(model_fn=model_fn, config=run_config) for epoch in range(n_epochs): estimator.train

How to control amount of checkpoint kept by tensorflow estimator?

阅读更多关于 How to control amount of checkpoint kept by tensorflow estimator?

I've noticed that the new Estimator API automatically saves checkpoints during the training and automatically restart from the last checkpoint when training was interrupted. Unfortunately it seems it only keeps last 5 check points. Do you know how to control the number of checkpoints that are kept during the training? Tensorflow tf.estimator.Estimator takes config as an optional argument, which can be a tf.estimator.RunConfig object to configure runtime settings.You can achieve this as follows: # Change maximum number checkpoints to 25 run_config = tf.estimator.RunConfig() run_config = run

Input multiple files into Tensorflow dataset

阅读更多关于 Input multiple files into Tensorflow dataset

问题 I have the following input_fn. def input_fn(filenames, batch_size): # Create a dataset containing the text lines. dataset = tf.data.TextLineDataset(filenames).skip(1) # Parse each line. dataset = dataset.map(_parse_line) # Shuffle, repeat, and batch the examples. dataset = dataset.shuffle(10000).repeat().batch(batch_size) # Return the dataset. return dataset It works great if filenames=['file1.csv'] or filenames=['file2.csv'] . It gives me an error if filenames=['file1.csv', 'file2.csv'] . In

TensorFlow Custom Estimator - Restore model after small changes in model_fn

阅读更多关于 TensorFlow Custom Estimator - Restore model after small changes in model_fn

问题 I am using tf.estimator.Estimator for developing my model, I wrote a model_fn and trained 50,000 iterations, now I want to make a small change in my model_fn , for example add a new layer. I don't want to start training from scratch, I want to restore all the old variables from the 50,000 checkpoint, and continue training from this point. When I try to do so I get a NotFoundError How can this be done with tf.estimator.Estimator ? 回答1: TL;DR The easiest way to load variables from a previous

How to use tensorflow debugging tool tfdbg on tf.estimator in Tensorflow?

阅读更多关于 How to use tensorflow debugging tool tfdbg on tf.estimator in Tensorflow?

问题 I am working with Tensorflow version 1.4, and I want to debug my train() function. In this link https://www.tensorflow.org/programmers_guide/debugger#debugging_tf-learn_estimators_and_experiments there is a way to do it for tf.contrib.learn Estimators , but I can not find a way to adapt it to the (new in version 1.4) tf.estimator . This is what I have tried: from tensorflow.python import debug as tf_debug # Create an estimator my_estimator = tf.estimator.Estimator(model_fn=model_fn, params

Tensorflow 2.0 Keras is training 4x slower than 2.0 Estimator

阅读更多关于 Tensorflow 2.0 Keras is training 4x slower than 2.0 Estimator

问题 We recently switched to Keras for TF 2.0, but when we compared it to the DNNClassifier Estimator on 2.0, we experienced around 4x slower speeds with Keras. But I cannot for the life of me figure out why this is happening. The rest of the code for both are identical, using an input_fn() that returns the same tf.data.Dataset, and using identical feature_columns. Been struggling with this problem for days now. Any help would be greatly greatly appreciated. Thank you Estimator code: estimator =

Getting free text features into Tensorflow Canned Estimators with Dataset API via feature_columns

阅读更多关于 Getting free text features into Tensorflow Canned Estimators with Dataset API via feature_columns

I'm trying to build a model that gives reddit_score = f('subreddit','comment') Mainly this is as an example i can then build on for a work project. My code is here . My problem is that i see that canned estimators e.g. DNNLinearCombinedRegressor must have feature_columns that are part of FeatureColumn class. I have my vocab file and know that if i was to just limit to the first word of a comment i could just do something like tf.feature_column.categorical_column_with_vocabulary_file( key='comment', vocabulary_file='{}/vocab.csv'.format(INPUT_DIR) ) But if i'm passing in say first 10 words from

Train Tensorflow model with estimator (from_generator)

阅读更多关于 Train Tensorflow model with estimator (from_generator)

I am trying train an estimator with a generator, but I want to feed this estimator with a package of samples for each iteration. I show the code: def _generator(): for i in range(100): feats = np.random.rand(4,2) labels = np.random.rand(4,1) yield feats, labels def input_func_gen(): shapes = ((4,2),(4,1)) dataset = tf.data.Dataset.from_generator(generator=_generator, output_types=(tf.float32, tf.float32), output_shapes=shapes) dataset = dataset.batch(4) # dataset = dataset.repeat(20) iterator = dataset.make_one_shot_iterator() features_tensors, labels = iterator.get_next() features = {'x':