I am developing a regression model to analyze walking styles. The dataset I am using to build the model is from 2 different sources, let's call them dataset A and dataset B.
Dataset A has a shape of (15000,6)
and dataset B has a shape of (15000,89)
. Both datasets are time-series data taken simultaneously for 5 minutes. I then trained the data using input data of shape (15000,89,1)
. The loss
and val_loss
generated from the training process were very small, but when I tested it using real data, the model was unable to predict the real data. Therefore, I think I am experiencing overfitting.
Then I tried adding new data to the input data. This new data is a sequence number from 1-50. I added this sequence number as an identity to the input data because the data I will generate will have 50 data in 1 second. I only added this sequence number to the input data. So the input data that was originally (15000,89,1)
became (15000,90,1)
. Then after adding this sequence number, I trained the new input data, and the result was better than before, and the model was able to predict real data. My question is, why is the technique of adding a sequence number to the input data able to avoid overfitting? Is the additional information of the sequence number part of data augmentation? If yes, please give me logical reasons and relevant references because I can't find relevant references and it is hard to find keywords for them.
you can refer to this question to understand the question in detail: https://datascience.stackexchange.com/questions/117711/adding-data-sequences-as-unique-data-on-dataset-for-regression-model?noredirect=1#comment118506_117711
(15000, 89, 1)
? $\endgroup$