I watched a video explaining how LSTM cells have very rudimentary feed-forward neural networks, basically a 2 layer input-output with no hidden layers.

Why don't LSTM cells have more complex neural networks before each gate, i.e. containing 1 hidden layer?

I would think that if you want more advanced gating decisions, that you would use at least 1 hidden layer to have more complex processing.

  • 3
    $\begingroup$ What is the video? Maybe it was considering the nodes themselves, as you can have multiple layers of LSTM nodes without problem, try it :) $\endgroup$
    – benbyford
    Jan 13 '20 at 20:57
  • 1
    $\begingroup$ The reason is likely because the benefits of increasing the number of hidden units in an LSTM outweigh (or are essentially the same as) the benefits from increasing the depth before each node. Each of these increase computation time, so you should always chose the one that has the largest benefit. This conclusion was likely come to simply through experimentation. I can not find any papers on this topic however, but I would guess it's been done as a small segment of a paper and was probably quickly shown to have minimal or no benefit $\endgroup$
    – Recessive
    Jan 14 '20 at 2:46
  • $\begingroup$ Mostly LSTM dealing with sequence of low-dimensional samples, like words (below 300 dims). For LSTM dealing with high-dimensional samples like images (much more then 10^3 dims) LSTM could be put on top of traditional feed-forward network like CNN $\endgroup$ Jan 15 '20 at 15:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.