# How do GPUs faciliate the training of a Deep Learning Architecture?

I would love to know in detail, how exactly GPUs help, in technical terms, in training the deep learning models.

To my understanding, GPUs help in performing independent tasks simultaneously to improve the speed. For example, in calculation of the output through CNN, all the additions are done simultaneously, and hence improves the speed.

But, what exactly happens in a basic neural network or in a LSTM type complex models in regard with GPU.

GPUs are able to execute a huge amount of similar and simple instructions (floating point operations like addition and multiplication) in parallel. In contrast to a CPU which is able to execute a few complex tasks sequentially very quick. Therefore GPUs are very good at doing vector & matrix operations.

If you look at the operations performed inside a single basic NN layer you will see that most of the operations are matrix-vector multiplications:

$$x_{i+1} = \sigma(W_ix_i + b_i)$$

where $$x_i$$ is the input vector, $$W_i$$ the matrix of weights, $$b_i$$ the bias vector in the $$i^{th}$$ layer, $$x_{i+1}$$ the output vector and $$\sigma(\cdot)$$ the elemtwise non-linear activation. The computational complexity here is governed by the matrix-vector multiplication. If you look at the architecture of a LSTM cell you will notice that inside of it are multiple such operations.

Being able to execute the matrix-vector operations quickly and efficiently in parallel will reduce execution time, this is where GPUs excel CPUs.

• This is quite insightful. You have pointed that CPUs are slow because they are capable of doing sequential tasks at best. But, to my understanding, I believe CPUs are slow because of less number of cores. A typical CPU constitutes 4 cores, whereas Tesla K80 (easily available general purpose GPU) has 4992 CUDA Cores. Hence, if we employ a CPU with comparable cores to K80, we might get same performance. What are your thoughts in this? – Anubhav Sachan Jun 16 '20 at 14:13
• @AnubhavSachan You wouldn't get the same performance, you'd get much better performance. CPUs are designed to have very fast single thread speed, but the cost/thread is much higher. So for \$1000 you could buy 4 CPU cores, but could buy 5000 GPU cores. It all depends on the situation which is better. But if you had the money, and wanted the absolute fastest performance, you'd saturate the calculations with CPU cores (by saturate I mean more cores wont help because there's only so much you can compute). – Recessive Jun 17 '20 at 3:00
• @Recessive Very nice addition to my answer! Thanks a lot! The cost/thread aspect is of course one of the main pain points here. I forgot to mention it. – Tinu Jun 17 '20 at 8:28
• Great answers! Thanks for your inputs. This question turned out to be a good discussion. – Anubhav Sachan Jun 17 '20 at 16:17