Is it possible and how trivial (or not) might it be (if possible) to retrain GPT-2 on time-series data instead of text?
2 Answers
Definitely! but at that point it would be training a transformer-encoder (gpt2's architecture) and not GPT2 because GPT2 is defined by the weights / training procedure / data it was trained and not the architecture, and I don't think it would transfer properly to time series.
In TS tasks, Transformers can capture long-range dependencies effectively through their self-attention mechanism, which can potentially lead to better forecasting performance compared to LSTMs. However, they are computationally more expensive than LSTMs.
-
$\begingroup$ It's worth noting that their self-attention mechanism is quadratic in time and space with regard to the input length. This may be intractable for very long sequences (>> 1k). However, solutions to this problem have already been proposed, such as the Sparse Transformer. $\endgroup$– Green绿色Apr 13 at 11:25