# Continuous sequence data with Transformer model

What is the right way to input continuous, temporal(time series) data into Transformer network. Assume we're using the basic TransformerBlock here.

Since data is continuous with no tokens, Token embedding can be directly skipped. How about positional encoding? I tried this example, removing Token embedding while keeping positional encoding but ended up with shape related errors. Skipping both token and positional encoding resulted in a network that runs and trains but results were relatively poor compared to LSTM benchmark with the same data.

I am unsure if positional encoding is still needed.

Overall, my question is, what is the proper way to process continuous sequence data such as time-series forecasting, using Transformer architecture?

Instead of using a token embedding you can use a linear layer. For an input of (10, 5, 4) - (sequence length, batch size, features) you can create a linear layer:

self.embedding_layer = nn.Linear(4, d_model)


Where d_model is the dimension of the input to the transformer.

PositionalEncoding is still needed so as to have a representation of time in the inputs.

src = self.embedding_layer(src)
src = self.pos_encoding_layer(src)
output = self.transformer(src)