# What size of neural networks can be trained on current consumer grade GPUs? (1060,1070,1080)

is it possible to give a rule of thumb estimate about the size of neural networks that are trainable on common consumer grade GPUs? For example:

The Emergence of Locomotion (Reinforcement) paper trains a network using tanh activation of the neurons. They have a 3 layer NN with 300,200,100 units for the Planar Walker. But they don’t report the hardware and time ...

But could a rule of thumb be developed? Also just based on current empirical results, so for example:

X Units using sigmoid activation can run Y learning iterations per h on a 1060.

Or using activation function a instead of b causes a n times decrease in performance.

If a student/researcher/curious mind is going to buy a GPU for playing around with these networks, how do you decide what you get? A 1060 is apparently the entry level budget option, but how can you evaluate if it is not smarter to just get a crappy netbook instead of building a high power desktop and spend the saved $on on-demand cloud infrastructure. Motivation for the question: I just purchased a 1060 and (clever, to ask the question afterwards huh) wonder if I should have just kept the$ and made a Google Cloud account. And if I can run my master thesis simulation on the GPU.

Usually the problem is to fit the model into video RAM. If it does not, you cannot train your model at all without big efforts (like training parts of the model separately). If it does, time is your only problem. But the difference in training time between consumer GPUs like the Nvidia 1080 and much more expensive GPU accelerators like the Nvidia K80 are not very large. Actually the best consumer cards are faster than GPU accelerators, but lack other properties like VRAM. Random comparisons and benchmarks: http://timdettmers.com/2017/04/09/which-gpu-for-deep-learning/, https://medium.com/@alexbaldo/a-comparison-between-nvidias-geforce-gtx-1080-and-tesla-p100-for-deep-learning-81a918d5b2c7

To calculate if your models fits into VRAM, you just approximate how much data and which hyperparameters you have (inputs, outputs, weights, layers, batch size, which datatype and so on).

• Good answer, but what kind of a model does not fit into a 6/8GB Memory? Let's say each batch of training data + the model. I have no idea how to calculate the size of the model but I cannot imagine a few thousands neurons and their weights can fill up Gigabytes of Memory. The Pretrained image recognition model from Google is a few hundred Megs tensorflow.org/tutorials/image_recognition and that can recognize more than most others out there. Apple handles it on-chip in the iPhone so the model itself must be much smaller than the T-data. And that doesn't need to be all in RAM right? – pascalwhoop Dec 16 '17 at 14:28
• I am often working with NLP models, mostly multi-layer RNNs, and if the vocabulary size is large, even 8 GB might not be enough. Remember that training usually needs more space than prediction (gradients take up space also). – C. Yduqoli Dec 18 '17 at 3:55

As a caveat, I’d suggest that unless you’re pushing up against fundamental technological limits, computation speed and resources should be secondary to design rationale when developing a neural network architecture.

That said, earlier this year I finished my MS thesis that involved bioinformatics analytics pipelines with whole genome sequencing data - that project took over 100,000 hours of compute time to develop according to our clusters job manager. When your on a deadline, resources can be a real constraint and speed can be critical.