0
$\begingroup$

Here is a confusion about the decoder in seq2seq. In each time-step in decoder, there are two outputs: 1.output 2.hidden. and this hidden state is used as the next input hidden state. this output is used as the next input state.

But when I use pytorch, I find 1.output 2.hidden are exactly the same thing. this means output=hidden, so what's the point to use two same thing twice?

in pytorch we can specify the length of rnn (corresponding to the number of words in a stentence, say N), but to implement the decoder, we often set the length to 1. and manullay do a loop N times. In this case, the formula "hidden=output" is true.

Maybe there is some misunderstanding here, however, I find now way to figure it out. Since I found no formula description(probability based) in my lecture slides. And I don't know how pytorch exactly works.

Besides, if anyone could give probability interpreter step by step, that would be fine. This confusion is related to another confusion. That is, what's meaning of hidden state? If it is understood to be a vector which contains info of all the sentence, why I can extract one predicted word from it by juse one matrix(the linear matrix connect to the output of the decoder in each step)?

Here is an example. suppose the output hidden state of encoder is $h_0$, and the first input of decoder is $x_1$, than the first output is $o_1$, first hidden output is $h_1$. and by the same method we define $x_i,o_i,h_i$. so we have $x_2=o_1,h_1=o_1$, and if we set the final output in each step as $y_i$, so suppose $y_i=Mo_i$, this means that we use the same $M$ to extract different things from $o_i$, this is a little strange, and another strange is that since $o_i=h_i$, we are using the same thing twice.

$\endgroup$

0

You must log in to answer this question.