New answers tagged

1

The original transformer is a feedforward neural network (FFNN)-based architecture that makes use of an attention mechanism. So, this is the difference: an attention mechanism (in particular, a self-attention operation) is used by the transformer, which is not just this attention mechanism, but it's an encoder-decoder architecture, which makes use of other ...


0

Instead of using a token embedding you can use a linear layer. For an input of (10, 5, 4) - (sequence length, batch size, features) you can create a linear layer: self.embedding_layer = nn.Linear(4, d_model) Where d_model is the dimension of the input to the transformer. PositionalEncoding is still needed so as to have a representation of time in the inputs....


Top 50 recent answers are included