问题
Basically I only reused code from iris utils and iris pipeline with minor change on serving input:
def _get_serve_tf_examples_fn(model, tf_transform_output):
model.tft_layer = tf_transform_output.transform_features_layer()
feature_spec = tf_transform_output.raw_feature_spec()
print(feature_spec)
feature_spec.pop(_LABEL_KEY)
@tf.function
def serve_tf_examples_fn(*args):
parsed_features = {}
for arg in args:
parsed_features[arg.name.split(":")[0]] = arg
print(parsed_features)
transformed_features = model.tft_layer(parsed_features)
return model(transformed_features)
def run_fn(fn_args: TrainerFnArgs):
...
feature_spec = tf_transform_output.raw_feature_spec()
feature_spec.pop(_LABEL_KEY)
inputs = [tf.TensorSpec(
shape=[None, 1],
dtype=feature_spec[f].dtype,
name=f) for f in feature_spec]
signatures = {
'serving_default':
_get_serve_tf_examples_fn(model, tf_transform_output).get_concrete_function(*inputs),
}
model.save(fn_args.serving_model_dir, save_format='tf', signatures=signatures)
the get_concrete_function() original input from iris codes is only a TensorSpec with dtype string. I already tried serving the model using the exact input but when I test the REST API I got a parsing error. So I tried to change the serving input so it can receive JSON input like this:
{"instances": [{"feat1": 90, "feat2": 23.8, "feat3": 12}]}
when I run the pipeline, the training was successful but then the error occurred when running the evaluator component. this is the latest logs:
INFO:absl:Using ./tfx/pipelines/toilet_native_keras/Trainer/model/67/serving_model_dir as candidate model.
INFO:absl:Using ./tfx/pipelines/toilet_native_keras/Trainer/model/14/serving_model_dir as baseline model.
INFO:absl:The 'example_splits' parameter is not set, using 'eval' split.
INFO:absl:Evaluating model.
INFO:absl:We decided to produce LargeList and LargeBinary types.
WARNING:tensorflow:5 out of the last 5 calls to <function recreate_function.<locals>.restored_function_body at 0x7fa7f0e44560> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.WARNING:tensorflow:6 out of the last 6 calls to <function recreate_function.<locals>.restored_function_body at 0x7fa7c77f8a70> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.
...
Traceback (most recent call last):
File "apache_beam/runners/common.py", line 1213, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam/runners/common.py", line 570, in apache_beam.runners.common.SimpleInvoker.invoke_process
File "/usr/local/lib/python3.7/site-packages/tensorflow_model_analysis/model_util.py", line 466, in process
result = self._batch_reducible_process(element)
File "/usr/local/lib/python3.7/site-packages/tensorflow_model_analysis/extractors/batched_predict_extractor_v2.py", line 164, in _batch_reducible_process
self._tensor_adapter.ToBatchTensors(record_batch), input_names)
AttributeError: 'NoneType' object has no attribute 'ToBatchTensors'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker.py", line 256, in _execute
response = task()
File "/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker.py", line 313, in <lambda>
lambda: self.create_worker().do_instruction(request), request)
File "/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker.py", line 483, in do_instruction
getattr(request, request_type), request.instruction_id)
File "/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker.py", line 518, in process_bundle
bundle_processor.process_bundle(instruction_id))
File "/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/bundle_processor.py", line 983, in process_bundle
element.data)
File "/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/bundle_processor.py", line 219, in process_encoded
self.output(decoded_value)
File "apache_beam/runners/worker/operations.py", line 330, in apache_beam.runners.worker.operations.Operation.output
...
File "apache_beam/runners/common.py", line 1294, in apache_beam.runners.common.DoFnRunner._reraise_augmented
File "/usr/local/lib/python3.7/site-packages/future/utils/__init__.py", line 446, in raise_with_traceback
raise exc.with_traceback(traceback)
File "apache_beam/runners/common.py", line 1213, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam/runners/common.py", line 570, in apache_beam.runners.common.SimpleInvoker.invoke_process
File "/usr/local/lib/python3.7/site-packages/tensorflow_model_analysis/model_util.py", line 466, in process
result = self._batch_reducible_process(element)
File "/usr/local/lib/python3.7/site-packages/tensorflow_model_analysis/extractors/batched_predict_extractor_v2.py", line 164, in _batch_reducible_process
self._tensor_adapter.ToBatchTensors(record_batch), input_names)
AttributeError: 'NoneType' object has no attribute 'ToBatchTensors' [while running 'ExtractEvaluateAndWriteResults/ExtractAndEvaluate/ExtractBatchPredictions/Predict']
...
WARNING:tensorflow:7 out of the last 7 calls to <function recreate_function.<locals>.restored_function_body at 0x7fa7f0273050> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.WARNING:tensorflow:8 out of the last 8 calls to <function recreate_function.<locals>.restored_function_body at 0x7fa7c77fc170> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes arg
I don't think evaluator component has anything to do with serving input function as it just compare to the newly trained model with a latest published model, but then where did I go wrong?
回答1:
So in the end I was mistaken about the evaluator component, or more appropriately if I address the TFMA instead. it indeed uses the serving input function defined in serving signatures. According to this link, the default signature used by the TFMA EvalConfig is "serving_default" which describes the serving model input to be serialized examples. That's why when I changed the input signature other than string, TFMA would raised as exception.
I think this signature is not meant to be used in serving the model through REST API and because the "serving_default" signature is still needeed and I am not in the mood with tinkering the EvalConfig, I created another signature which would receive the JSON input that I want. For tht to work, I need to make another function decorated by @tf.function. That's all. I hope my answer will help people who struggle with similar problems.
来源：https://stackoverflow.com/questions/64530885/tfx-pipeline-error-while-executing-tfma-attributeerror-nonetype-object-has-n