What are the typical sizes of practical/commercial artificial neural networks?

Question

I'm interested in artificial neural networks (ANN) and I wonder how big ANNs in practical use are, for example, Tesla Autopilot, Google Translate, and others.

The only thing I found about Tesla is this one:

"A full build of Autopilot neural networks involves 48 networks that take 70,000 GPU hours to train. Together, they output 1,000 distinct tensors (predictions) at each timestep."

It seems like most companies don't publish clear information about their ANN sizes. I really can't find anything detailed on this subject.

Is there any information about the size of big practical/commercial ANNs that include something like the amount of neurons/connections/layers etc.?

I'm looking for a few examples in this scale with more precise information on the size of the neural networks.

Answer 1

NLP Domain

You can easily find such open-source neural networks in NLP applications that have been published by Companies like Google. For example, in BERT models, you can see the BERT-Base has the following specifications:

BERT-Base, Multilingual Cased: 104 languages, 12-layer, 768-hidden, 12-heads, 110M parameters

You can find more data about other versions of BERT in the same link.

Another example is GPT models, like GPT-3:

All GPT-3 models use the same attention-based architecture as their GPT-2 predecessor. The smallest GPT-3 model (125M) has 12 attention layers, each with 12x 64-dimension heads. The largest GPT-3 model (175B) uses 96 attention layers, each with 96x 128-dimension heads.

Image Procssing Domain

Another useful domain for your expectation is image processing tasks such as image classification. Pretrained models such as VGG, ResNet, and Inception. These are mostly used for image classification tasks in different companies, and you can find their specification many where. For example for VGG-16, we can see the followings:

Speech Processing

Another practical domain is Auto-Speech Recognition or ASR in short. One of the renowned models in this context is DeepSpeech(2) by Baido Research center. For example, you can find some info like the number of its parameters and its structure in this github link.

Summing up

Note that one regular metric to measure the size of neural networks is "the number of parameters" of the network that is required to be learned in the training phase. Hence, you can compare the size of models even between cross domains by knowing their number of parameters (instead of going to more details about the number of hidden layers and their types). Although, sometimes the length (number of layers) and height (number of neurons in each layer) of the network are very important in the matter of performance and capability of the network.

Answer 2

I hope this helps. Disclaimer: the info is extracted from Computer Vision at Tesla, though aditional references may be needed....

• Tesla is using 8 cameras per vehicle fused with radars.
• To detect road lanes, vehicles, pedestrians etc., Tesla have to run at least 50 neural networks simultaneously.
• The neural networks are trained using PyTorch. Each image (for each camera I guess) is of dimension (1280, 960, 3). RGB image it seems.
• Regarding the neural network architecture, "the backbone is a modified ResNet 50", that is, a convolutional network.
• The network is 50 layers deep. The network depth is defined as the largest number of sequential convolutional or fully connected layers on a path from the input layer to the output layer. In total, ResNet-50 has 177 layers. The ResNet-50 has over 23 million trainable parameters
• Every camera is processed through a single neural network. Then everything is combined into a middle neural network.

Answer 3

The size of the model depends on the domain. I am currently working with a model that is used for real time inference on an embedded device. Speed of computation is critical.

The model size is a 5 layer CNN, about 700k parameters and it's about 12MB in size on disk.

Answer 4

Use the benchmarked algorithms or research papers will be a good start. Addition to that use the open sourced Bert GPT 2 like architectures is a good start.