I am training LSTM Nets with Keras on a small mobile GPU. The speed on GPU is slower then on CPU. I found some articles that say that it is hard to train LSTMs (RNNs) on GPUs because the training cannot be parallelized.

What is your experience? Is LSTM training on large GPUs like 1080 Ti faster then on CPU?


From nvidia www (https://developer.nvidia.com/discover/lstm):

Accelerating Long Short-Term Memory using GPUs

The parallel processing capabilities of GPUs can accelerate the LSTM training and inference processes. GPUs are the de-facto standard for LSTM usage and deliver a 6x speedup during training and 140x higher throughput during inference when compared to CPU implementations. cuDNN is a GPU-accelerated deep neural network library that supports training of LSTM recurrent neural networks for sequence learning. TensorRT is a deep learning model optimizer and runtime that supports inference of LSTM recurrent neural networks on GPUs. Both cuDNN and TensorRT are part of the NVIDIA Deep Learning SDK.

  • $\begingroup$ Ok. Thanks. Did you try it by yourself? $\endgroup$ – Dieshe Jul 9 '18 at 9:16
  • $\begingroup$ @Dieshe: No, my activity is in agi area $\endgroup$ – pasaba por aqui Jul 9 '18 at 9:23

I found that there are cuDNN accelerated cells in Keras for example: https://keras.io/layers/recurrent/#cudnnlstm They very fast. The normal LSTM cells are faster on CPU then on GPU. Also see here for a comparisem: https://wiki.eniak.de/ml/geschwindigkeitsvergleich_keras_lstm_und_cudnnlstm

  • $\begingroup$ All the well known deep learning frameworks have gpu accelaration facility for that matter (not only keras). And most of them use cuDNN underneath but exposing different API calls for the user.. $\endgroup$ – varsh Jul 22 '18 at 6:08
  • $\begingroup$ Well - the above was just an example. Because normal LSTM Cells are slower... $\endgroup$ – Dieshe Jul 23 '18 at 8:52
  • 3
    $\begingroup$ the second link is broken $\endgroup$ – gizzmole Feb 19 '19 at 15:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.