1
$\begingroup$

Say I have a machine learning model trained on a laptop and I then want to embed/deploy the model on a microcontroller. How can I do this?

I know that TensorflowLite Micro generates a C header to be added in the project and then be embedded, but every example I read shows how it is done with neural networks, which seems legit as TensorFlow is primarily used for deep learning.

But how can I do the same with any type of model, like the ones there is in scikit-learn? So, I'm not interested in necessarily doing this with TensorflowLite Micro, but I'm interested in the general approach to solve this problem.

$\endgroup$
1
  • $\begingroup$ Please, rather than editing your post to add an answer to your own question, add a formal answer below and provide the relevant details from that blog post in it. $\endgroup$ – nbro Jan 18 at 11:43
4
$\begingroup$

There are a few possible approaches to deploying a ML model to a microcontroller.

The main limiting factor to deployment on microcontollers is that ML models are usually a representation of a set of parameters that are intended to be used as input to a prediction algorithm alongside a new datapoint. Most such models assume the presence of an accompanying library that implements the algorithm in question. However, a microcrontoller may use an exotic chip architecture, or have very several or unusual resource constraints that prevent these standard libraries from being deployed easily.

Presumably you will already have some way to get input into your microcontroller and to program it in order to call some function that you can write. If not, you will need to first figure out how to do that, and the right methods depend on your microcontroller. A common approach is to write assembly code or code in a very limited subset of C or another language. An alternative is to find a distribution of an interpreter for another language (e.g. Java, Python) that has been compiled to work on your chip. Either way, you will need some way to program the chip.

Presuming you can program the chip, you have two fundamental challenges in deploying the model:

  1. Most models are trained with very wide floating point numbers for their parameters. For example, 128-bit numbers may be used. On a standard computing environment, the CPU or GPU will be equipped to perform operations on wide datatypes efficiently. On a microcontroller, you may be limited to 8-bit or 16-bit integers. To work with your model parameters in an environment like this, you will need to either make the parameters smaller (usually by rounding them to fit in a much smaller numeric format, a process called "quantization"), or by finding or writing software that can simulate the operations you want (probably addition and multiplication) on a large datatype that is represented as a collection of smaller datatypes. The first approach may make the model perform poorly. The second may make model prediction very slow.

  2. You need an implementation of the algorithm. Some algorithms like linear regression, linear discriminant analysis, or even decision trees are extremely easy to implement prediction for, and may require only addition, multiplication, and/or comparison. You might be able to write these yourself in a simple subset of C, or even in assembly (for example, prediction with linear regression should be just a simple loop). Other algorithms, like deep neural networks, may contain more complex operations, and may contain many such operations performed in complicated sequences. For these, you generally will need to find an distribution of a library that implements the algorithms, or compile one yourself. Compiling one yourself will require setting up a build toolchain for your specific microcontroller, and can be quite involved.

$\endgroup$
2
$\begingroup$

If the library running the model can be compiled for your microcontroller, then you can run your model on that microcontroller.

If you train using one library and deploy using another library, you possible can convert your model to that library: ONNX.

Some library links on Edge Computing in ML:

To speed things up specific hardware is used: GPU's of course, but also Tensor Processing Units.

One can manually transfer an NN trained model to C code (paper), but there are also compilers:

XLA and Glow support x86-64 and ARM64. The NNCG approach generates C code and thus should be more readily portable to general MCU's (paper).

Further links: adge-ai, stackoverlow question

$\endgroup$
2
  • $\begingroup$ Isn't ONNX only for neural networks? The OP is interested in deploying any ML model (so not just NNs). By the way, welcome to AI SE :) $\endgroup$ – nbro Jan 17 at 14:22
  • $\begingroup$ ONNX is for NN exchange. ML can be seen as a very broad field. Any C code can do ML. As such a one for all exchange/deploy format would mean quite a bit of coordination and standardization. As the previous answer said, ML models normally are split into two parts: data relative to a library. Every model has its own library. One needs to have that availability on the target platform. $\endgroup$ – Roland Puntaier Jan 17 at 21:57
2
$\begingroup$

I found:

These seems to partly fit my needs. But i am surprised that i cannot find something more general that either convert Python to C or to object file with ML support (to be used in C projects). Indeed, once trained, ML algo are "just" a bunch of additions/multiplications/comparisons.

It would help to be able to convert scikit-learn pipeline for instance, as ML projects are rarely composed of only a single ML model taking raw data. But it seems that EdgeAI/TinyML is mainly focused on compiling deep learning models when it comes to deploy ML models on "bare-metal" microcontrollers.

Thanks for your answers btw, it helps.

$\endgroup$
1
$\begingroup$

One can use simpler approach with deepC compiler and convert exported onnx model to c++.

Check out simple example at deepC compiler sample test Compile onnx model for your target machine

Checkout mnist.ir

Step 1:

Generate intermediate code

% onnx2cpp mnist.onnx

Step 2:

Optimize and compile

% g++ -O3 mnist.cpp -I ../../../include/ -isystem ../../../packages/eigen-eigen-323c052e1731/ -o mnist.exe

Step 3:

Test run

% ./mnist.exe

Here is a link to YouTube Video for elaborate instructions.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.