How much can the addition of new features improve the performance?

Question

How much can the addition of new features improve the performance of the model during the optimization process?

Let's say I have a total of 10 features. Suppose I start the optimisation process using only 3 features.

Can the addition of the 7 remaining ones improve the performance of the model (which, you can assume, might already be quite high)?

Answer 1

It depends on the used network as well as the feeding mechanism but let's give an example;

When working with LSTM, giving the time data (as an integer sequence) in addition to the time-series data(coming from features) dramatically increases the performance of the network.

[$X_{0}$,$X_{1}$, ...] $\rightarrow$ [[$X_{0}$,$t_{0}$],$[X_{1}$,$t_{1}$], ...]

If you go and look for the kaggle competition winner's notebooks, they do also create additional features based on the featured data.

Let's assume that the performance is already quite high on the three features so that you can predict those three features with high reliability. It would only make sense to increase the number of features if you would like to predict additional features!

Answer 2

The optimization of 10 features, as opposed to optimization of 3 features, will converge more slowly.

Let's say I have a total of 10 features. Suppose I start the optimisation process using only 3 features. Can the addition of the 7 remaining ones improve the performance of the model (which, you can assume, might already be quite high).

The answer might be no.

• The accuracy and reliability of the convergence toward ground truth (a formalized objective used to guide the optimization process) when split into groups of three and seven features may be better or worse than when left as a group of ten.
• Except in rare cases, the results will not be the same. The likelihood of identical results is so low it may never happen in the world in the next century except when conditions are arranged solely to cause it to occur.

Why then do may approaches group the dimensions of the result and converge on groups of axes, then another, then another, and back again to the first, iterating until the convergence goal is reached? This approach is used to reduce the time and computing resources used to reach the optimal. As the problem complexity increases, the use of groupings in this way is more common.