3
$\begingroup$

Say I have a machine learning model trained on a laptop and I then want to embed/deploy the model on a microcontroller. How can I do this?

I know that TensorflowLite Micro generates a C header to be added in the project and then be embedded, but every example I read shows how it is done with neural networks, which seems legit as TensorFlow is primarily used for deep learning.

But how can I do the same with any type of model, like the ones there is in scikit-learn? So, I'm not interested in necessarily doing this with TensorflowLite Micro, but I'm interested in the general approach to solve this problem.

$\endgroup$
1
  • $\begingroup$ Please, rather than editing your post to add an answer to your own question, add a formal answer below and provide the relevant details from that blog post in it. $\endgroup$
    – nbro
    Jan 18, 2021 at 11:43

4 Answers 4

6
$\begingroup$

There are a few possible approaches to deploying a ML model to a microcontroller.

The main limiting factor to deployment on microcontollers is that ML models are usually a representation of a set of parameters that are intended to be used as input to a prediction algorithm alongside a new datapoint. Most such models assume the presence of an accompanying library that implements the algorithm in question. However, a microcrontoller may use an exotic chip architecture, or have very several or unusual resource constraints that prevent these standard libraries from being deployed easily.

Presumably you will already have some way to get input into your microcontroller and to program it in order to call some function that you can write. If not, you will need to first figure out how to do that, and the right methods depend on your microcontroller. A common approach is to write assembly code or code in a very limited subset of C or another language. An alternative is to find a distribution of an interpreter for another language (e.g. Java, Python) that has been compiled to work on your chip. Either way, you will need some way to program the chip.

Presuming you can program the chip, you have two fundamental challenges in deploying the model:

  1. Most models are trained with very wide floating point numbers for their parameters. For example, 128-bit numbers may be used. On a standard computing environment, the CPU or GPU will be equipped to perform operations on wide datatypes efficiently. On a microcontroller, you may be limited to 8-bit or 16-bit integers. To work with your model parameters in an environment like this, you will need to either make the parameters smaller (usually by rounding them to fit in a much smaller numeric format, a process called "quantization"), or by finding or writing software that can simulate the operations you want (probably addition and multiplication) on a large datatype that is represented as a collection of smaller datatypes. The first approach may make the model perform poorly. The second may make model prediction very slow.

  2. You need an implementation of the algorithm. Some algorithms like linear regression, linear discriminant analysis, or even decision trees are extremely easy to implement prediction for, and may require only addition, multiplication, and/or comparison. You might be able to write these yourself in a simple subset of C, or even in assembly (for example, prediction with linear regression should be just a simple loop). Other algorithms, like deep neural networks, may contain more complex operations, and may contain many such operations performed in complicated sequences. For these, you generally will need to find an distribution of a library that implements the algorithms, or compile one yourself. Compiling one yourself will require setting up a build toolchain for your specific microcontroller, and can be quite involved.

$\endgroup$
4
$\begingroup$

I found:

These seems to partly fit my needs. But i am surprised that i cannot find something more general that either convert Python to C or to object file with ML support (to be used in C projects). Indeed, once trained, ML algo are "just" a bunch of additions/multiplications/comparisons.

It would help to be able to convert scikit-learn pipeline for instance, as ML projects are rarely composed of only a single ML model taking raw data. But it seems that EdgeAI/TinyML is mainly focused on compiling deep learning models when it comes to deploy ML models on "bare-metal" microcontrollers.

Thanks for your answers btw, it helps.

$\endgroup$
5
  • $\begingroup$ Is there any particular preprocessing / feature extraction things you have in mind, apart from the model itself? $\endgroup$
    – Jon Nordby
    Dec 30, 2021 at 23:13
  • $\begingroup$ say the signal comes online. The pipeline could be: the signal is acqquired for n samples (windowed) and some features are extracted from it (mean, var, and other dedicated ones). An ACP can then be used to reduce the dimentionality, and then the reduced vector is fed to a classifier. This is a classic approach actually, but i did not find a simple way to covert the whole pipeline (at the time. But i did not search recently). But i will need to go back to it in the coming weeks. $\endgroup$
    – Hattori
    Jan 4 at 12:45
  • $\begingroup$ Yes the overlapped windows and feature calculation is something one almost always have to do with time-series. However this is not entirely standardized even in Python, tends to be custom per project. But I am interested in implementing this in emlearn, with C implementations tailored for microcontroller use (streaming, low-memory, no dynamic allocations). $\endgroup$
    – Jon Nordby
    Jan 4 at 14:54
  • $\begingroup$ What is ACP? Did you mean PCA dimensionality reduction? $\endgroup$
    – Jon Nordby
    Jan 4 at 14:56
  • $\begingroup$ oops indeed, PCA ! ACP is the french name. I will further explore what is possible to do in the coming weeks, and come back here if anything relevant to add. $\endgroup$
    – Hattori
    Jan 5 at 19:58
2
$\begingroup$

If the library running the model can be compiled for your microcontroller, then you can run your model on that microcontroller.

If you train using one library and deploy using another library, you possible can convert your model to that library: ONNX.

Some library links on Edge Computing in ML:

To speed things up specific hardware is used: GPU's of course, but also Tensor Processing Units.

One can manually transfer an NN trained model to C code (paper), but there are also compilers:

XLA and Glow support x86-64 and ARM64. The NNCG approach generates C code and thus should be more readily portable to general MCU's (paper).

Further links: adge-ai, stackoverlow question

$\endgroup$
2
  • $\begingroup$ Isn't ONNX only for neural networks? The OP is interested in deploying any ML model (so not just NNs). By the way, welcome to AI SE :) $\endgroup$
    – nbro
    Jan 17, 2021 at 14:22
  • $\begingroup$ ONNX is for NN exchange. ML can be seen as a very broad field. Any C code can do ML. As such a one for all exchange/deploy format would mean quite a bit of coordination and standardization. As the previous answer said, ML models normally are split into two parts: data relative to a library. Every model has its own library. One needs to have that availability on the target platform. $\endgroup$ Jan 17, 2021 at 21:57
1
$\begingroup$

One can use simpler approach with deepC compiler and convert exported onnx model to c++.

Check out simple example at deepC compiler sample test Compile onnx model for your target machine

Checkout mnist.ir

Step 1:

Generate intermediate code

% onnx2cpp mnist.onnx

Step 2:

Optimize and compile

% g++ -O3 mnist.cpp -I ../../../include/ -isystem ../../../packages/eigen-eigen-323c052e1731/ -o mnist.exe

Step 3:

Test run

% ./mnist.exe

Here is a link to YouTube Video for elaborate instructions.

Fur details, read more at https://link.medium.com/IhbcJzi4jgb

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .