8
$\begingroup$

I am training LSTM neural networks with Keras on a small mobile GPU. The speed on the GPU is slower than on the CPU. I found some articles that say that it is hard to train LSTMs (and, in general, RNNs) on GPUs because the training cannot be parallelized.

Is this true? Is LSTM training on large GPUs, like 1080 Ti, faster than on CPUs?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
9
$\begingroup$

From Nvidia www (https://developer.nvidia.com/discover/lstm):

Accelerating Long Short-Term Memory using GPUs

The parallel processing capabilities of GPUs can accelerate the LSTM training and inference processes. GPUs are the de-facto standard for LSTM usage and deliver a 6x speedup during training and 140x higher throughput during inference when compared to CPU implementations. cuDNN is a GPU-accelerated deep neural network library that supports training of LSTM recurrent neural networks for sequence learning. TensorRT is a deep learning model optimizer and runtime that supports inference of LSTM recurrent neural networks on GPUs. Both cuDNN and TensorRT are part of the NVIDIA Deep Learning SDK.

Improve this answer
$\endgroup$
0
6
$\begingroup$

I found that there are cuDNN accelerated cells in Keras, for example, https://keras.io/layers/recurrent/#cudnnlstm. They are very fast. The normal LSTM cells are faster on CPU than on GPU.

Improve this answer
$\endgroup$
2
  • $\begingroup$ All the well known deep learning frameworks have gpu accelaration facility for that matter (not only keras). And most of them use cuDNN underneath but exposing different API calls for the user.. $\endgroup$
    – varsh
    Commented Jul 22, 2018 at 6:08
  • $\begingroup$ Well - the above was just an example. Because normal LSTM Cells are slower... $\endgroup$
    – Dieshe
    Commented Jul 23, 2018 at 8:52

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .