0
$\begingroup$

I am developing a regression model to analyze walking styles. The dataset I am using to build the model is from 2 different sources, let's call them dataset A and dataset B.

Dataset A has a shape of (15000,6) and dataset B has a shape of (15000,89). Both datasets are time-series data taken simultaneously for 5 minutes. I then trained the data using input data of shape (15000,89,1). The loss and val_loss generated from the training process were very small, but when I tested it using real data, the model was unable to predict the real data. Therefore, I think I am experiencing overfitting.

Then I tried adding new data to the input data. This new data is a sequence number from 1-50. I added this sequence number as an identity to the input data because the data I will generate will have 50 data in 1 second. I only added this sequence number to the input data. So the input data that was originally (15000,89,1) became (15000,90,1). Then after adding this sequence number, I trained the new input data, and the result was better than before, and the model was able to predict real data. My question is, why is the technique of adding a sequence number to the input data able to avoid overfitting? Is the additional information of the sequence number part of data augmentation? If yes, please give me logical reasons and relevant references because I can't find relevant references and it is hard to find keywords for them.

you can refer to this question to understand the question in detail: https://datascience.stackexchange.com/questions/117711/adding-data-sequences-as-unique-data-on-dataset-for-regression-model?noredirect=1#comment118506_117711

$\endgroup$
3
  • $\begingroup$ First thing is I believe your problem is "underfitting". Second is, can you describe in details how you concat A and B to have shape (15000, 89, 1)? $\endgroup$ Commented Mar 8, 2023 at 11:03
  • $\begingroup$ no, I not concat the data. so the input data is (15000,89,1) $\endgroup$ Commented Mar 8, 2023 at 11:05
  • $\begingroup$ in other hand I want predict dataset A using dataset B,so from (15000,89) will predict (15000,6) $\endgroup$ Commented Mar 8, 2023 at 11:06

1 Answer 1

0
$\begingroup$

You are doing a technique called "feature engineering", where we manually add new features to the dataset that tell something about the data, so that the model is able to learn better. Every changes you make to the input data is already called Feature Engineering, even something small as adding a new number in the input data.

Kaggle even has a dedicated course for Feature Engineering. Also, it has been empirically proven to help the model, such as in here, here, and here.

$\endgroup$
2
  • $\begingroup$ but in my case the data will work when I adding number of sequence each array $\endgroup$ Commented Mar 8, 2023 at 11:26
  • $\begingroup$ the ilustration as in this picture i.sstatic.net/Hd891.png $\endgroup$ Commented Mar 8, 2023 at 11:27

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .