问题
TensorFlow website claims that Quantization provides up to 3x lower latency on mobile devices: https://www.tensorflow.org/lite/performance/post_training_quantization
I tried to verify this claim, and found that Quantized models are 45%-75% SLOWER than Float models in spite of being almost 4 times smaller in size. Needless to say, this is very disappointing and conflicts with Google's claims.
My test uses Google's official MnasNet model: https://storage.googleapis.com/mnasnet/checkpoints/mnasnet-a1.tar.gz
Here is the average latency based on 100 inference operations on a freshly rebooted phone:
- Pixel 2: float=81ms, quant=118ms
- Moto E: float=337ms, quant=590ms
- LG Treasure: float=547ms, quant=917ms
My test app measures the timing for only one method (tfLite.runForMultipleInputsOutputs). The results are very consistent (within 1% across multiple executions).
I am hoping to see some comments from the Tensorflow team or anybody who can share their metrics. The numbers above are based on image classifier model. I also tested an SSD MobileNetV2 object detector with similar results (quantized model being substantially slower).
来源：https://stackoverflow.com/questions/55958129/tensorflow-lite-quantization-fails-to-improve-inference-latency