0
$\begingroup$

I am developing a regression model to analyze walking styles. The dataset I am using to build the model is from 2 different sources, let's call them dataset A and dataset B.

Dataset A has a shape of (15000,6) and dataset B has a shape of (15000,89). Both datasets are time-series data taken simultaneously for 5 minutes. I then trained the data using input data of shape (15000,89,1). The loss and val_loss generated from the training process were very small, but when I tested it using real data, the model was unable to predict the real data. Therefore, I think I am experiencing overfitting.

Then I tried adding new data to the input data. This new data is a sequence number from 1-50. I added this sequence number as an identity to the input data because the data I will generate will have 50 data in 1 second. I only added this sequence number to the input data. So the input data that was originally (15000,89,1) became (15000,90,1). Then after adding this sequence number, I trained the new input data, and the result was better than before, and the model was able to predict real data. My question is, why is the technique of adding a sequence number to the input data able to avoid overfitting? Is the additional information of the sequence number part of data augmentation? If yes, please give me logical reasons and relevant references because I can't find relevant references and it is hard to find keywords for them.

you can refer to this question to understand the question in detail: https://datascience.stackexchange.com/questions/117711/adding-data-sequences-as-unique-data-on-dataset-for-regression-model?noredirect=1#comment118506_117711

Improve this question
$\endgroup$
3
  • $\begingroup$ First thing is I believe your problem is "underfitting". Second is, can you describe in details how you concat A and B to have shape (15000, 89, 1)? $\endgroup$
    – Minh-Long Luu
    Mar 8 at 11:03
  • $\begingroup$ no, I not concat the data. so the input data is (15000,89,1) $\endgroup$
    – stack offer
    Mar 8 at 11:05
  • $\begingroup$ in other hand I want predict dataset A using dataset B,so from (15000,89) will predict (15000,6) $\endgroup$
    – stack offer
    Mar 8 at 11:06

1 Answer 1

Reset to default
0
$\begingroup$

You are doing a technique called "feature engineering", where we manually add new features to the dataset that tell something about the data, so that the model is able to learn better. Every changes you make to the input data is already called Feature Engineering, even something small as adding a new number in the input data.

Kaggle even has a dedicated course for Feature Engineering. Also, it has been empirically proven to help the model, such as in here, here, and here.

Improve this answer
$\endgroup$
2
  • $\begingroup$ but in my case the data will work when I adding number of sequence each array $\endgroup$
    – stack offer
    Mar 8 at 11:26
  • $\begingroup$ the ilustration as in this picture i.stack.imgur.com/Hd891.png $\endgroup$
    – stack offer
    Mar 8 at 11:27

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .