Determining optimal data size for generalization in transformer encoders, particularly for Time-Series signal data

Question

I'm currently experimenting with training a model that employs a single transformer encoder on time-series signal data. Despite having a relatively small dataset of around 50 examples, each with a sequence length of approximately 1000, the model seems to excel at understanding and memorizing these examples. However, I'm concerned about its generalization capabilities given the limited amount of data.

I'm wondering: How much data is typically required for a transformer encoder to generalize well, especially in the context of time-series signal data? Is there a recommended range or guideline for the amount of training data that can help ensure better generalization performance? Additionally, are there specific strategies or techniques that can enhance generalization in transformer models when working with small datasets?

Any insights, experiences, or references would be greatly appreciated. Thank you!

Answer 1

In order for a neural network to generalize well, it has to be trained on tons and tons of data, otherwise if the model is shown very few samples of the training data, it will be biased towards these few samples and it will perform poorly on unseen data for sure.

Transformers are data hungry models and won't get generalized with few samples (50 in your case)

Best suggestion for your use case is to use some pretrained model that has been trained extensively on the time series data. Best example would be timesfm from google research. Pick this pre-trained model as the starting point for your weights and fine tune the model with your custom training samples.