I'm currently working on a regression problem and I have 10 inputs/attributes.

What should I do if there are correlations between different features of the input data? Does the correlation between inputs affect the performance (e.g. accuracy) of the model?


Non-correlation does not imply independence, that is, if two features are not correlated (i.e. zero correlation), it does not mean that they are independent. But (non-zero) correlation implies dependence (see https://stats.stackexchange.com/q/113417/82135 for more details). So, if you have non-zero correlation between two features, it means they are dependent. If they are dependent, then one feature gives you information about the other and vice-versa: in a certain way, one of the two is, at least partially, redundant.

Unnecessary features might not affect the performance (e.g. the accuracy) of a model. However, if you reduce the number of features, the learning process might actually be faster.

You may want to try some dimensionality reduction technique, in order to reduce the number of features.

  • For a linear regression model, the features act as x values for the equation y =mx + c. I don't think that the correlation between features would bring change in the performance of the model.
  • Linear regression is only the measure of linear association among y and x values.
  • But the main thing is that the prediction will only be linear. That is the function which establishes a relationship between the input and the output will be linear. So, you need to perform various experiments with your features.

Strictly, correlations between any two or more inputs do not affect a model used in regression during use, however, the model may emerge from the analysis of correlations between input features. This is an important AI concept and one of the times when philosophy is directly applicable to science and technology.

To create a model that may explain observed phenomena, the patterns in observations must be recognized and a plausible relationship between variables representing the dimensions of the observation must be determined. That plausible relationship must be tested by correlating data to the model.

However, once the model is stable in the research and development sense, it can be used to control or predict. At that time, the very patterns that caused the emergence of the model must be decoupled (at least in the batch or real time processing) from the model for the model to have use.

A simple case is that observations of current, resistance, and potential voltage led to the model $V = IR$, but that relation is useless to electrical engineers if it is questioned because of newly acquired data. If the readings from a meter do not match the model, it is the readings that are questioned.

Correlations in a set of $(V, I, R)$ samples were discovered in a noisy quantum domain of electron travel. These correlations were discovered during feature extraction too. The combination of assignment of features macroscopic electrical events and the model that relates those features led to the understanding of proportionality or inverse proportionality in the relationship between any pair of those three dimensions.

Said another way, the model emerged through correlation of pairs (when the third variable was held constant) to $x$ or $1/x$ models. After the model gained acceptance in the scientific community, the correlations between input dimensions were not directly used. Neither were the correlations between the model and the original data of early researchers into electricity reexamined.

Today, the correlation between the model and the two known values and the unknown value is the only correlation in common use. A formula is used instead of regression for two reasons.

  • Two inputs are known to be proportional or inversely proportional to the output of that model (depending on which is the dependent variable when put to use).
  • The expectation of accuracy is low, so only one or two samples may be used.

When the demand for accuracy increases, regression may become important to gain significant digits in quantification of the unknown property. For instance, an accurate resistance value may require several current-voltage pairs. This is a model that reduces to closed form. Other models do not, so no formula can be generated. Convergence strategies that use iteration or recursion, including those leveraged by regression, artificial networks, and other AI components become useful.

To answer the final question in the question body above, the performance of the model depends on correlation between past observations and the model, but performance in terms of accuracy and reliability at run time only depend on the qualities of the model and the quality of its integration into the AI system. The quality of its integration will also affect speed and conservation of computing resources.

This question touches on statistics because regression is a statistical method. It touches on data science because the philosophic aspect of science pronounced by Newton, Maxwell, Planck, von Neumann, and Heisenberg plays out in the realm of data. This question is directly related to AI because the application of intelligence is clear in the development of a model and then the trusting of it as a generalization to address specific cases.

Interestingly, a paradigm shift occurs when trusted models fail under an increasingly annoying array of conditions. These anomalies then lead to the seeking of a new model to which the larger, more contemporary data set correlates with greater accuracy and reliability. As paradigms shift to serve curiosity and technology, human beings must exhibit the mental features that roughly fall under the umbrella term intelligence. But this kind of intelligence begins with doubt and unlearning, not learning.

The subset of intelligence features that facilitates scientific discovery falls under another umbrella term, cognition. During a paradigm shift, the correlation between input variables again falls under scrutiny because doubt in the accepted model is re-introduced.

Response to Comments

The concept of cognition belongs in any comprehensive answer to this question, since correlation correlates with causality. This is central to AI and central to the reason why many regression methods produce correlation indicators. In the analysis of time series for the purposes of prediction, management, or control (which are the only practical reasons to do any statistics at all) the shift in time is significant.

$$ \mathcal{M}_{\Delta t_1, \Delta t_2, ..., \Delta t_X} \Bigg[ \mathcal{C}(\mathcal{Y}, f(P, \mathcal{X}')) \, \land \, \Big(\forall \; i \in \{1, 2, ..., X\} \, \land \, \mathcal{x}'_i(t) = \mathcal{x}_i(t) - \Delta t_i \Big) \Bigg] \; \text{,} $$

where $\mathcal{M}$ is maximization, $\Delta t_i$ is the shift in time that produces the maximum correlation, $\mathcal{C}$ is the correlation mechanism, $\mathcal{Y}$ is the desired result of applying model $f$, parameterized by $P$, to the component-wise shifted values of the features time series $\mathcal{X}'$.

In this case, the optimal shifts of time in the input parameters is evidence of but not proof of causality $\mathcal{C}$. It is itself a correlation $\mathcal{C}'$.

$$ \mathcal{C}' \Big( \text{Pr} (x_{\alpha} \implies x_{\beta}), \mathcal{C} \Big) $$

The doubt and certainty indicated by $\mathcal{C}'$ is the doubt and certainty in cognition and the basis for belief. If we ignore this abstraction, then the answer to these questions are superficially conformed to basic statistics but absent of the concept of cognition.

My question is what to do if there are correlations among different features of the input data? Does correlation b/w inputs affect the performance of Model?

  • 1
    $\begingroup$ What the heck are you talking about? Can you write answers where you focus on answering the OP's question(s), or is it too difficult for you? For example, what the heck does "The subset of intelligence features that facilitates scientific discovery falls under another umbrella term, cognition." have to do with the OP's question? $\endgroup$ – nbro Jan 27 '19 at 13:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.