I would love to know in detail, how exactly GPUs help, in technical terms, in training the deep learning models.
To my understanding, GPUs help in performing independent tasks simultaneously to improve the speed. For example, in calculation of the output through CNN, all the additions are done simultaneously, and hence improves the speed.
But, what exactly happens in a basic neural network or in a LSTM type complex models in regard with GPU.
GPUs are able to execute a huge amount of similar and simple instructions (floating point operations like addition and multiplication) in parallel. In contrast to a CPU which is able to execute a few complex tasks sequentially very quick. Therefore GPUs are very good at doing vector & matrix operations.
If you look at the operations performed inside a single basic NN layer you will see that most of the operations are matrix-vector multiplications:
$$x_{i+1} = \sigma(W_ix_i + b_i)$$
where $x_i$ is the input vector, $W_i$ the matrix of weights, $b_i$ the bias vector in the $i^{th}$ layer, $x_{i+1}$ the output vector and $\sigma(\cdot)$ the elemtwise non-linear activation. The computational complexity here is governed by the matrix-vector multiplication. If you look at the architecture of a LSTM cell you will notice that inside of it are multiple such operations.
Being able to execute the matrix-vector operations quickly and efficiently in parallel will reduce execution time, this is where GPUs excel CPUs.
