New answers tagged transformer
1
The original transformer is a feedforward neural network (FFNN)-based architecture that makes use of an attention mechanism. So, this is the difference: an attention mechanism (in particular, a self-attention operation) is used by the transformer, which is not just this attention mechanism, but it's an encoder-decoder architecture, which makes use of other ...
0
Instead of using a token embedding you can use a linear layer. For an input of (10, 5, 4) - (sequence length, batch size, features) you can create a linear layer:
self.embedding_layer = nn.Linear(4, d_model)
Where d_model is the dimension of the input to the transformer.
PositionalEncoding is still needed so as to have a representation of time in the inputs....
Top 50 recent answers are included
