1
$\begingroup$

Is it possible and how trivial (or not) might it be (if possible) to retrain GPT-2 on time-series data instead of text?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

Definitely! but at that point it would be training a transformer-encoder (gpt2's architecture) and not GPT2 because GPT2 is defined by the weights / training procedure / data it was trained and not the architecture, and I don't think it would transfer properly to time series.

Improve this answer
$\endgroup$
1
  • $\begingroup$ GPT-2 is a decoder, not an encoder. $\endgroup$
    – Green绿色
    Apr 13 at 11:20
0
$\begingroup$

In TS tasks, Transformers can capture long-range dependencies effectively through their self-attention mechanism, which can potentially lead to better forecasting performance compared to LSTMs. However, they are computationally more expensive than LSTMs.

Improve this answer
$\endgroup$
1
  • $\begingroup$ It's worth noting that their self-attention mechanism is quadratic in time and space with regard to the input length. This may be intractable for very long sequences (>> 1k). However, solutions to this problem have already been proposed, such as the Sparse Transformer. $\endgroup$
    – Green绿色
    Apr 13 at 11:25

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .