7
$\begingroup$

I have a Remaining Useful Life (RUL) prediction problem that I want to solve. When I added two or more features as inputs to my ANN, the accuracy of my ANN has been decreased. More precisely, I've added features like RMS or KURTOSIS (or both). I was expecting the system to improve, but it is getting worse.

Why might this be happening? What are the potential reasons for this degradation in performance?

I know that when we added more nodes in layers (like hidden layers), overfitting can happen. Would that be related to my problem: using more than two features?

$\endgroup$
1
  • $\begingroup$ A simple case is if the new features noisy. $\endgroup$
    – SpiderRico
    Mar 22, 2020 at 20:14

1 Answer 1

2
$\begingroup$

Additional features can also cause overfitting if they have low or misleading information.

Consider the following problem:

$X = [1, 3, 3, 4, 5]$, $Y = [1, 3, 4, 4, 5]$.

Suppose that the real dataset was generated from the relationship:

$Y = X$, with a probability of 0.2 of adding or subtracting 1.

A reasonable model estimate is $Y = X$. Note that no model can fit this data perfectly, because the two 3 inputs map to different outputs.

Now, suppose we add a new feature: a random number between $0$ and $10$: $W = [1 ,5, 2, 6, 3]$

It may not be obvious, but a sufficiently deep and broad neural network can learn a new function:

$g(W) = 1$ if $W = 2,4,7,8,9,$ or $0$.

$g(W) = 0$ otherwise.

and define a new prediction: $Y = X - g(W)$.

This happens to produce a perfect fit on the training data. However, it will perform extremely poorly on new data (like a test set), because it has learned a meaningless pattern out of random noise. Coincidentally, it will be wrong on about 50% of samples, while our first model will be wrong on only 20% of samples.

$\endgroup$
2
  • $\begingroup$ Maybe you mean Y = [1, 2, 3, 4, 5]? $\endgroup$
    – nbro
    Oct 22, 2019 at 23:27
  • $\begingroup$ @nbro No, I intended for there to be some error in the original set, but on second look, I need to add some extra data for this example to make sense. $\endgroup$ Oct 22, 2019 at 23:30

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .