Is it possible and how trivial (or not) might it be (if possible) to retrain GPT-2 on time-series data instead of text?
$\begingroup$
$\endgroup$
$\begingroup$
$\endgroup$
Definitely! but at that point it would be training a transformer-encoder (gpt2's architecture) and not GPT2 because GPT2 is defined by the weights / training procedure / data it was trained and not the architecture, and I don't think it would transfer properly to time series.
