3
$\begingroup$

I am Reading "Supervised Sequence Labelling with Recurrent Neural Networks" written by Alex Graves to try to understand LSTM networks and I am a bit confused about the equations.

Specifically, what I am confused about is the term "state". When used in an equation (section 4.5.2), it says:

enter image description here

I know that some system can be in a state, for example, due to the setup of values of different nodes in a graph. But how can a state be described in the case of a neural network and how can the equation above be explained other than that it is the state (or states of several timesteps as in recurrent neural networks) of a neural network?

Improve this question
$\endgroup$
3
  • $\begingroup$ I'm actually quite baffled by these equations. The state at time t requires the state at time t+1... $\endgroup$
    – BlueMoon93
    Jun 1, 2017 at 9:21
  • $\begingroup$ @BlueMoon93 When talking about the dependencies of different values in different timesteps in recurrent neural networks, it has been claimed (if I understood correctly) that BPTT should be the same as the chain rule in calculus... however I don't think the term "state" is as intuitive as e.g. "output" which is the actual value that has been received after passing something through an activation function. $\endgroup$
    – Stephen Johnson
    Jun 1, 2017 at 9:24
  • $\begingroup$ I have found this blog post to be very helpful in understanding LSTMs:colah.github.io/posts/2015-08-Understanding-LSTMs $\endgroup$
    – Aiden Grossman
    Jun 1, 2017 at 14:13

1 Answer 1

Reset to default
3
$\begingroup$

So the equation that you mentioned is used during the backward pass in which back proppogation is performed in order to make the neural network more accurate. I think you are talking about the state during the forward pass which is completely different. In the forward pass, the neural network is simply run in order to evaluate or it is simply used as a model. The repeating module in long short term memory networks looks like this: LSTM repeating module As you can see there are many different parts to this module. There are three main parts. The first is the forget gate layer. This layer tells the cell state or the line running across the stop what to keep. The cell state is kept by this line:enter image description here The entire network is based off of manipulating this cell state in order to get accurate results. The equation that you mentioned was related to backprop which is used to train the neural network. This is related to the cell state because it is used to calculate it during the backwards pass. @BlueMoon93 mentioned that this equation has t+1 one in it, but this is because as the recurrent neural network propogates backwards through each module the time goes from high to low. So to conclude, the cell state in an LSTM is one of the vectors that the neural network modifies based on input.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .