I'm reading Sequence to Sequence Learning with Neural Networks and there's a thing that I couldn't quite grasp.

Paper says the encoder outputs a vector to be fed to the decoder. More precisely

Our method uses a multilayered Long Short-Term Memory (LSTM) to map the input sequence to a vector of a fixed dimensionality, and then another deep LSTM to decode the target sequence from the vector

However, when I look at the diagram:

enter image description here

there's no such vector here. What I understand from this diagram is decoder RNN takes the weights of the last encoder cell as an input.

Which one is correct? can you explain?

Stanford notes put it as

The final hidden state of the cell will then become C

So, is there no vector?

  • $\begingroup$ w here does not represent weights, in case you were thinking it does. $\endgroup$ Nov 11 '21 at 8:55

That drawing it's a bit oversimplified. Check this blog for a better explanation and implementation details. I'll refer to the image they have to answer:

  • the yellow boxes represent embedding layers, required to convert words in numbers
  • the green boxes represent the unfolded encoder
  • the red box represent the context vector, i.e. the vector you're looking for. Note that is just the final vector you obtain by applying the encoder to a sequence of words. For this reason some people prefer to draw directly a line to the decoder part, without drawing the final vector explicitly.
  • the blue boxes represent the unfolded decoder.
  • the purple boxes represent the linear layer used to predict a final word for the decoder hidden state.

enter image description here

  • $\begingroup$ The blog is reversing the input order and the paper also says that they reversed the order. But at the diagram you mentioned it's not in reverse order. Is it just an error or intentional do you have any ideas? For example it supposed to be morgen guten I guess. $\endgroup$
    – J.Smith
    Nov 11 '21 at 10:29
  • $\begingroup$ yes, it's suppose to be <sos> morgen guten <eos>. I guess it's probably an error, or they left the correct order in the diagram just for readability. $\endgroup$ Nov 11 '21 at 10:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.