Machine Learning hardware usage in embedded applications

Question

I've been reading a lot about hardware development and implementation for AI/ML, mainly about Deep Learning, and I have a question about its usage. From what I understand, there are 2 stages for DL: first is training and second is inference. The first is often done on GPUs because of their massive parallelism capabilities among other things, and inference, while can be done on GPUs, it's not used that much, because of power usage, and because the data presented while inferring are much less so the full capabilities of GPUs won't be much needed. Instead FPGAs and CPUs are often used for that.

My understanding also is that a complete DL system will have both, a training system and an inferring system.

My question is that: are both systems required on the same application? Let's assume an autonomous car or an application where visual and image recognition is done, will it have both training system to be trained and an inference system to execute? Or it has only the inference system and will communicate with a distant system which is already trained and has built a database?

Also, if the application has both systems, will it have a big enough memory to store the training data? Given that it can be a small system and memory is ultimately limited.

Answer 1

To answer your question: Training and inference are usually completed on two separate systems you are right in knowing that training of deep neural networks is usually done on GPUs and that inference is usually done on CPUs. However, training and inference are almost always done on two separate systems. The main workflow for many data scientists today is as follows: 1. Create and establish all hyper-parameters for a model such as a deep neural network 2. Train the deep neural network using a GPU 3. Save the weights that training on the GPU established so that the model can be deployed. 4. Code the model in a production application with the optimal weights found in training.

So as you can see from this workflow training and inference are done in two completely separate phases.

However, in some specific cases training and inference are done on the same system. For example, if you are using a Deep Neural Network to play video games than you may have the neural network train and infer on the same system. This would lead to more efficiently because it would allow the model to continuously learn.

To answer your question on memory, the only applications where inference and training are done in the same application have a lot of memory available(think dual GPU dual CPU 128gb of RAM workstations) whereas applications that have a limited amount of memory only use inference such as embedded applications.

Answer 2

Embedded artificial intelligence is a booming area of development. The hardware and software basics are in place, but the research into how to design mature home and industrial products is only beginning to approach the vision cast for it.

Deep learning lovers will point to some papers that theoretically approach machine control using policy based action decision modalities. Those actually working on machine control know that the convergence of these systems is not workable yet for field operational machines. The more general the machine must operate, the more unworkable a machine learning solution is.

Aaeon Up produces GPU and computer on a board products using Intel chip sets that support embedded AI development and NVidia and Qualcom are expected to follow.

Inference is the noun that corresponds to infer. Deep networks don't ever infer anything during operation after trained. What they do is do. The terms in use don't include inference.

• Production
• In the field
• Deployed
• In situ

Inference is usually the term used in the predicate logic or rules system sub-field of AI. Deep networks do not, by themselves, contain logic other than the logic in the CPU, FPU, or GPU that performs arithmetic. After training, the are essentially programmed DSPs (digital signal processors).

(Some training systems can be reentered to further training and some deep designs allow concurrent training and operation, but learning and doing are not fully integrated in deep learning models yet.)

Power consumption, heat dissipation, size, and weight are issues in embedded systems, which makes memory usage and thrift a challenge. This is typical, and not just for AI apps. In aeronautics, the effects of gravity and acceleration compound the challenge greatly.

The data science methods that work well for FaceBook, Google, Amazon, Lexis Nexis, and Pentaho are not designed for light-weight, low-power applications. In autonomous vehicle systems, whether they are ground vehicles, submerged vehicles, or flight vehicles makes huge difference in these respects.

---

Are both systems required on the same application?

In vehicles in general, portions of the function can be learned at different points of time.

1. During a vendor's design of a family of models
2. During a vendor's finalization of a particular model with its accessories
3. During the quality control of a particular ID numbered vehicle before it is shipped
4. During the first few months of usage, where the vehicle may learn the particular usage patterns and environment of the buyer
5. During the remaining life of the vehicle to produce continuous operational improvement and adaptation to normal ware and maintenance processes

---

An autonomous car or an application where visual and image recognition is done, will it have both training system to be trained and an inference system to execute? Or [does] it [have] only the [deployed trained run time system] and communicate with a distant system which is already trained and has built a database?

For any military, commercial, or delivery network vehicle, requiring communication with a home base leaves open a number of security risks and possible attack vectors. Patch systems with proper security is probable, and several vehicle manufacturers have had firmware updates that are installed every time the vehicle is serviced.

Training data can be compressed using lz4 or simply by packing data as has been done in satellite communications since the 1970s. The memory is challenged more by the training that can only occur during field use of the product. Embedded product designers have developed over the last half century a large array of useful techniques to address the challenge, and more will be developed along side the real time AI technology. The two disciplines will emerge together.

Answer 3

Deep learning seems mostly to be a buzzword for what is essentially a neural network. You train with a data set to recognize a pattern, then input new data which is then classified by the trained network.

So you train a neural network with 10 different kinds of animals using thousands of pictures. Then you show the network say 100 new images and have the network "guess" what each animal is.

The point here is that training a neural network would required code for feedback that an application using the trained network would not need. So an application just using the trained network would be a bit more streamlined than an application which could allow additional training data.

What is missing is the ability for machine learning to form hierarchical or otherwise interrelated rules from the trained network so that the trained network is closer to being able to rationally explain why the classification works. Going back to the 10 animal trained network, without formulation of rules during training, there is no practical way for the network to reveal why any of the animals was classified the way it was.