0
$\begingroup$

In a time series regression problem I'm predicting "change" rather than the actual intended value i.e Instead of:

time, feature 1, feature 2, ....
2020, 1, 3.3
2021, 1.5, 5.2
2022, 1.3, 6.1
now, y_pred_1, y_pred_2, ....

I'm feeding the model and predicting percent changes:

2020 0, 0
2021 0.5, 0.57 # 5.2 / 3.3 - 1 = 0.57 percent change
2022, -0.13, 0.17

Now a problem that I'm facing is that when I introduce a new feature of a much larger magnitude and variance my model diverges during training:

# new feature
2020, 3000
2021 4
2022 600

# Which is transformed to percent changes
2020, 0
2021, 0.99
2022, 149 # oof

I tried mean normalizing that particular feature before transforming it to the percent change domain but it still diverges. Any ideas what else I can try to introduce this new feature to the inputs? What are some best practices for such contrasted inputs?

To get a general idea of the model if it has something to do with this:

The model consists of 5 parallel flows that get added together at the end. Each flow has two pairs of CNNs with different kernel sizes followed by 1-3 LSTMs. The output shape of each flow is the same so they can be added together. Then after adding it's followed by two fully connected layers predicting the change for the given features.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

The problem as you already pointed out is in the high change variance of the new feature. Normalising using the mean does not impact the variance since it's merely a linear transformation, still good to keep all feature in the same range but not impacting the variance at all.

What you need is a non linear transformation that change the magnitude in change variance without introducing artefacts. The most simple thing to try would be to just take the log of the new feature. Since you're working with time series another possibility could be applying a moving average, exploring different time windows to see how much the feature will be smoothed out.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks! If I used log as a normalization would it not generally "suppress" the sensitivity of the model to the changes of that feature? Or is that the tradeoff I have to accept? $\endgroup$
    – Darren Rahnemoon
    Apr 12 at 23:26
  • $\begingroup$ Also is there any point for choosing a specific base of the log function given the variance or just better off with natural log? $\endgroup$
    – Darren Rahnemoon
    Apr 12 at 23:28

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .