What is the right way to input continuous, temporal(time series) data into Transformer network. Assume we're using the basic TransformerBlock here.
Since data is continuous with no tokens, Token embedding can be directly skipped. How about positional encoding? I tried this example, removing Token embedding while keeping positional encoding but ended up with shape related errors. Skipping both token and positional encoding resulted in a network that runs and trains but results were relatively poor compared to LSTM benchmark with the same data.
I am unsure if positional encoding is still needed.
Overall, my question is, what is the proper way to process continuous sequence data such as time-series forecasting, using Transformer architecture?