If my understanding of an LSTM is correct then the output from each LSTM unit is the hidden state from that layer. For the final layer if I wanted to predict e.g. a scalar real number, would I want to add a dense layer with 1 neuron or is it recommended to have a final LSTM layer where the output has just one hidden unit (i.e. the output dimension of the final hidden state is 1)? If we didn't add the dense layer, then the output from the hidden layer I believe would be between (-1,1), if you use the traditional activations in the LSTM unit.

Apologies if I've used wrong terminology, there seems to be some inconsistency with LSTM's when going between literature and definition in TensorFlow etc.

  • $\begingroup$ Have you tried to have a look at some reference implementations? Here the author seems to use a dense layer with 1 unit after the last LSTM layer. $\endgroup$ – nbro Jun 10 at 22:31
  • 1
    $\begingroup$ Yes, I have used LSTM's in the past and added a dense layer with 1 unit on. This was from a walkthrough I found -- I was more questioning whether you have to do this, although intuitively I think that you do, especially if it is was in a scenario like I had described. e: I just clicked the link and that was the walkthrough I had followed haha. $\endgroup$ – David Ireland Jun 10 at 22:34
  • $\begingroup$ Your question is actually interesting. It's been a while I've used LSTMs, so I cannot answer it now. Of course, if the output of the last LSTM layer is a number in the range $[-1, 1]$ and you need a number in $\mathbb{R}$, then you probably need to fix that (maybe by changing the activation function), if you want to try not to use the dense layers. $\endgroup$ – nbro Jun 10 at 22:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.