5
$\begingroup$

I'm interested in artificial neural networks (ANN) and I wonder how big ANNs in practical use are, for example, Tesla Autopilot, Google Translate, and others.

The only thing I found about Tesla is this one:

"A full build of Autopilot neural networks involves 48 networks that take 70,000 GPU hours to train. Together, they output 1,000 distinct tensors (predictions) at each timestep."

It seems like most companies don't publish clear information about their ANN sizes. I really can't find anything detailed on this subject.

Is there any information about the size of big practical/commercial ANNs that include something like the amount of neurons/connections/layers etc.?

I'm looking for a few examples in this scale with more precise information on the size of the neural networks.

$\endgroup$
0
2
$\begingroup$

NLP Domain

You can easily find such open-source neural networks in NLP applications that have been published by Companies like Google. For example, in BERT models, you can see the BERT-Base has the following specifications:

BERT-Base, Multilingual Cased: 104 languages, 12-layer, 768-hidden, 12-heads, 110M parameters

You can find more data about other versions of BERT in the same link.

Another example is GPT models, like GPT-3:

All GPT-3 models use the same attention-based architecture as their GPT-2 predecessor. The smallest GPT-3 model (125M) has 12 attention layers, each with 12x 64-dimension heads. The largest GPT-3 model (175B) uses 96 attention layers, each with 96x 128-dimension heads.

Image Procssing Domain

Another useful domain for your expectation is image processing tasks such as image classification. Pretrained models such as VGG, ResNet, and Inception. These are mostly used for image classification tasks in different companies, and you can find their specification many where. For example for VGG-16, we can see the followings:

enter image description here

Speech Processing

Another practical domain is Auto-Speech Recognition or ASR in short. One of the renowned models in this context is DeepSpeech(2) by Baido Research center. For example, you can find some info like the number of its parameters and its structure in this github link.

Summing up

Note that one regular metric to measure the size of neural networks is "the number of parameters" of the network that is required to be learned in the training phase. Hence, you can compare the size of models even between cross domains by knowing their number of parameters (instead of going to more details about the number of hidden layers and their types). Although, sometimes the length (number of layers) and height (number of neurons in each layer) of the network are very important in the matter of performance and capability of the network.

$\endgroup$
1
$\begingroup$

I hope this helps. Disclaimer: the info is extracted from Computer Vision at Tesla, though aditional references may be needed....

$\endgroup$
1
  • $\begingroup$ As interesting as this are the hyperparameters and data used to train these networks. Any info on this? $\endgroup$ May 25 at 7:38
0
$\begingroup$

The size of the model depends on the domain. I am currently working with a model that is used for real time inference on an embedded device. Speed of computation is critical.

The model size is a 5 layer CNN, about 700k parameters and it's about 12MB in size on disk.

$\endgroup$
0
$\begingroup$

Use the benchmarked algorithms or research papers will be a good start. Addition to that use the open sourced Bert GPT 2 like architectures is a good start.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.