Why don't people always use TensorFlow Lite, if it doesn't decrease the accuracy of the models?

Question

I have been exploring edge computation for AI, and I came across multiple libraries or frameworks, which can help to convert the model into a lite format, which is suitable for edge devices.

• TensorFlow Lite will help us to convert the TensorFlow model into TensorFlow lite.
• OpenVino will optimise the model for edge devices.

Questions

1. If we have a library to optimise the model for edge devices (e.g. TensorFlow Lite), after conversion, could it make the accuracy decrease?

2. If not, then why do people prefer don't always use e.g. TensorFlow Lite?

Answer 1

This partly answer to question 1. There is no general rule concerning accuracy or size of the model. It depends on the training data and the processed data. The lightest is your model compared to the full accuracy model the less accurate it will be. I would run the lite model on test data and compare to the accuracy of the full model to get an exact measure of the difference.

Tensor flow has different options to save the "lite" model (optimized in size, latency, none and default).

The following mostly answer question 2.

Tensor flow lite is intented to provide the ability to use the model to on line predict only and load the model not to train the model.

On the other hand Tensor flow is used to build (train) the model off line.

If your edge platform support any of the binding language provided for TensorFlow (javascript, java/kotlin, C++, python) you can use Tensorflow for prediction. The accuracy or speed options you might have selected to create the model will not be affected whether you use Tensor Flow or Tensor Flow Lite. Typically Tensor flow lite can be used on mobile devices (Ios, Android). There are other supported target, see this link

Answer 2

I have explored the AI for edge devices. My findings for tflite model.

1. TFLite is just the tool suite to convert TFModel into TFLite.
2. TFLite optimizes the model for edge or embedded devices using quantization techniques.

Quantization dramatically reduces both the memory requirement and computational cost of using neural networks.

Answer in brief:

1. When we optimize the model definitely they are faster to take inference but it will impact the accuracy. Quantization leads to bit accuracy loss in smaller networks.
2. TFLite is increasing the model performance but we have to pay a cost in terms of accuracy that's why we have both TFLite for edge computation and TFModel as well.