1
$\begingroup$

Considering the architecture of encoder and decoder in transformer as shown below: enter image description here

  • Does each input token after self attention mechanism (z1,z2,z3,...)is passed to it's specific separate Feed forward neural network or does all the Z's are stacked into one and then passed to single FFNN?
  • If all the Z's are stacked into one, then how the difference in shapes of different inputs is taken care
  • If every z has its own Feed forward neural network, how in practical it is implemented with arbitrary input length?
$\endgroup$

1 Answer 1

0
$\begingroup$
  • the z’s are stacked and then passed to a single feed forward layer
  • the difference in shapes of inputs is resolved by padding to the dimension of the model
$\endgroup$
1
  • $\begingroup$ Hi @Arjun Reddy! If possible, please add some supporting information to this answer to help other readers understand why the answer is correct. Thanks! $\endgroup$
    – DeepQZero
    May 18, 2023 at 15:16

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .