1
$\begingroup$

I'm trying to read this paper describing Google's LSTM architecture for machine translation from 2016. However, I'm getting stuck as certain things are described too vaguely for me.

enter image description here

This is a picture of the architecture. I just want to understand the encoder for now (the left block). The pink and green blocks are LSTM cells, I understand.

  1. Are these LSTM cells the same kind that are described on Wikipedia? Because those have two outputs (the "mask" $c_t$ and the state $h_t$) and three inputs. Which arrow is which in the Google diagram?

  2. Where are the recurrent connections going? Does the hidden state of each LSTM cell go into the cell on the right at the next timestep? The cell above? Into itself? All three? What about the mask vector?

  3. What goes where? That is, where do the inputs enter? The text $x_3, x_2, ...$ at the bottom of the encoder block is not clear to me. So the different tokens of the input go into different cells? What if there are more tokens in the sentence than cells?

Improve this question
$\endgroup$
  • $\begingroup$ Please, ask one question per post. You're asking many questions here. I suggest that you edit this post to ask one question. Additional questions should be asked in their separate post. I know that you're questions are all about the encoder part, but I still think you're asking many questions here. $\endgroup$ – nbro♦ Nov 21 '20 at 17:50
  • $\begingroup$ @nbro I don't think an answer to any one of those questions would really make any sense if they weren't in the context of an answer to the others as well. The question really is "could someone please write up a detailed account of what exactly this architecture is" - the four questions are really just "hints" to give a better sense of what such a description might look like. $\endgroup$ – Jack M Nov 21 '20 at 18:44
  • $\begingroup$ The type of question: "Describe me all the details of this model X" is not appropriate for SE. It's too broad. Either 1. you focus on a specific problem/question that you have or 2. you ask about the general idea behind a model (this latter is fine because you're not asking for all the details). So, I still suggest that you ask only one question at a time. The reasons are described in this answer. $\endgroup$ – nbro♦ Nov 21 '20 at 19:32
  • $\begingroup$ So, chances are that only a few people know the answer to all the questions in your post and maybe those people do not even visit this site. If you restrict yourself to specific/simpler problems/questions, chances are that someone could give a try to answer that simpler question, even without being an expert on the topic. $\endgroup$ – nbro♦ Nov 21 '20 at 19:34
  • 1
    $\begingroup$ @nbro Okay, I think I can just about split out point (4) as its own question. $\endgroup$ – Jack M Nov 23 '20 at 11:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.