I have a time series sequence with 10 million steps. In step $t$, I have a 400 dimensional feature vector $X_t$ and a scalar value $y_t$ which I want to predict during inference time and I know during the train time. I want to use a transformer model. I have 2 questions:
- If I want to embed the 400 dimensional input feature vector into another space before feeding into the transformer, what are the pros and cons of using let's say 1024 and 64 for the embedding space dimension? Should I use a dimension more than 400 or less?
- When doing position embedding, I cannot use a maximum position length of 10 million as that blows up the memory. What is the best strategy here if I want to use maximum position length of 512? Should I chunk the 10 million steps into blocks of size 512 and feed each block separately into the transformer? If so, how can I connect the subsequent blocks to take full advantage of parallelization while keeping the original chronological structure of the sequence data?
