Positional Encoding of Time-Series features

Question

I’m trying to use a Transformer Encoder I coded with weather feature vectors which are basically 11 features about the weather in the dimension [batch_size, n_features].

I have a data point per day, so this is a time-series but there are no evidences of the time in my vectors and I know I must use positional encoding to tell my network “this vector is from the 21/05/2021, this one is from the 20/04/2019, etc”.

The usual positional encoding with sin & cos used in NLP doesn’t seem to fit my problem as it encodes the position relative to other words in the sentence and my features are independant values (the temperature of the day doesn’t come after the amount of rain for instance). Order of the data matters between two different inputs but not within the features.

I don’t really know how to do encode my feature vectors for my transformer encoder to get something like positional_encoded = original_vector + date_encoding which I think could be great but I could also just add the day of year as a feature - would it be enough for a transformer?

What would be the best way to do so?

Answer 1

I think your input should have shape [batch_size, time_points, n_features] which would correspond to [batch_size, sentance_length, embedding_dim] in the standard transformer.

The positional encoding would encode the ordering of the samples in time.

The normal positional encoding I imagine is the standard thing to do. The only different thing to do would be to have a mpl transform the features into embedding vectors which then the positional encoding would be applied to.

If you wanted to add the time as a feature I would embed it in the half unit circle. Normalize time to the interval [0,1] then do torch.view_as_real(torch.exp(PI * normalized_time * 1j)) and append this to you features. The reason is that you want the dot product to be a similarity measure of the time embedding.