4
$\begingroup$

What is the right way to input continuous, temporal (time-series) data into the Transformer? Assume we're using the basic TransformerBlock here.

Since data is continuous with no tokens, Token embedding can be directly skipped. How about positional encoding? I tried this example, removing Token embedding while keeping positional encoding but ended up with shape-related errors. Skipping both token and positional encoding resulted in a network that runs and trains but results were relatively poor compared to the LSTM benchmark with the same data.

I am unsure if the positional encoding is still needed.

Overall, my question is, what is the proper way to process continuous sequence data, such as time-series, using the Transformer architecture?

$\endgroup$

1 Answer 1

3
$\begingroup$

Instead of using a token embedding you can use a linear layer. For an input of (10, 5, 4) - (sequence length, batch size, features) you can create a linear layer:

self.embedding_layer = nn.Linear(4, d_model)

Where d_model is the dimension of the input to the transformer.

PositionalEncoding is still needed so as to have a representation of time in the inputs.

src = self.embedding_layer(src)
src = self.pos_encoding_layer(src)
output = self.transformer(src)
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .