What is the right way to input continuous, temporal(time series) data into Transformer network. Assume we're using the basic TransformerBlock here.

Since data is continuous with no tokens, Token embedding can be directly skipped. How about positional encoding? I tried this example, removing Token embedding while keeping positional encoding but ended up with shape related errors. Skipping both token and positional encoding resulted in a network that runs and trains but results were relatively poor compared to LSTM benchmark with the same data.

I am unsure if positional encoding is still needed.

Overall, my question is, what is the proper way to process continuous sequence data such as time-series forecasting, using Transformer architecture?


Instead of using a token embedding you can use a linear layer. For an input of (10, 5, 4) - (sequence length, batch size, features) you can create a linear layer:

self.embedding_layer = nn.Linear(4, d_model)

Where d_model is the dimension of the input to the transformer.

PositionalEncoding is still needed so as to have a representation of time in the inputs.

src = self.embedding_layer(src)
src = self.pos_encoding_layer(src)
output = self.transformer(src)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.