I'll give an example on height and weight. Weight and height are correlated, but it's not necessarily the case that a tall person weighs more or that someone who weighs more is a tall person - and multicollinearity, as I understand it, is when we can say for sure that if we change one variable, the other variable changes as well (like if we change the entire advertising budget, the TV budget will change too, if we distribute all the money evenly), and that's the problem. And what does that have to do with the fact that correlation does not imply causation? All the videos and articles I've watched say, "Multicollinearity is the correlation between your variables, blablabla, and the problem with it is that if you change one variable, you change the other variable. And so, we can't calculate the unique effect of the variable" (i.e., as I understand it, they are emphasising the mandatory nature of this co-variation). And then they say to me "well, you know, we can't predict the second variable from the first, even knowing the first, it can be different, and it doesn't necessarily have to change with the first. It's not causation, it's not dependence, it's the influence of one variable on another" - and I ask - what's the difference?
2 Answers
You're raising a good point about the complexity of multicollinearity and its relationship with correlation and causality
1. Correlation
=> Correlation refers to a statistical relationship between two variables. If two variables are correlated, it means that changes in one are associated with changes in the other. However, this association does not imply that one causes the other to change.
=> Example of it Height and weight are correlated. Taller people tend to weigh more, but not in every case. The correlation just shows that there's a general trend, but it does not mean being tall causes a person to weigh more.
2. Causation
=> Causation means that one variable directly affects the other. If variable A causes variable B, changes in A will lead to changes in B by some predictable mechanism.
Example: If a person eats more calories, they tend to gain weight. Here, increased calorie intake directly causes weight gain.
The phrase "correlation does not imply causation" means: Just because two variables are correlated doesn't mean one causes the other. There could be other factors (confounders) at play, or the relationship could be coincidental.
Here The phrase "correlation does not imply causation" means:
Just because two variables are correlated doesn't mean one causes the other. There could be other factors (confounders) at play, or the relationship could be coincidental.
3. Multicollinearity
Multicollinearity occurs when two or more independent (predictor) variables in a regression model are highly correlated, meaning one variable can be predicted from the other(s) with a high degree of accuracy.
Problem: In a regression model, we want to measure the unique effect of each independent variable on the dependent variable. When multicollinearity is present, it becomes difficult to determine which variable is actually affecting the dependent variable, because the predictor variables are so strongly correlated that they "move together."
What's different here?
=> Multicollinearity is about relationships between independent variables, not about causality between them. The problem arises because, in regression, we assume that each predictor variable has its own independent influence on the outcome. When predictors are highly correlated, we can't easily distinguish which variable is responsible for the variation in the outcome.
=> Correlation vs. Causality: When we say "correlation does not imply causation," we're talking about how two variables can move together without one necessarily causing the other. Multicollinearity, however, deals with the issue that two (or more) predictor variables are so correlated that it's hard to tell their independent contributions to the model, but this doesn't imply a direct cause-effect relationship between them.
Example:
=> Let's say we're building a regression model to predict weight (dependent variable) using height and shoe size as predictors. Height and shoe size are likely correlated. If they're highly correlated (multicollinearity), it becomes hard to tell whether weight is being influenced more by height or shoe size. Even though they're correlated, this doesn't mean that height causes shoe size or vice versa, or that there's causality between them.
=> Multicollinearity creates issues in estimating regression coefficients, not in implying causality between predictors. It clouds the unique influence of each predictor but doesn’t necessarily deal with cause and effect. Does that help clarify the distinction?
Well, maybe someone could use this. I didn't understand why everyone says “if one thing changes, another thing changes” when we have correlation, not causation, and even with correlation = 1 we can't guarantee that one thing will change with the other. So, we assume “based on the old law of correlation, the data is likely to go like this”. That is, we say “okay, okay, it's not going to change 100%, but we've seen them change together, so it's likely they'll change this time too”. It reminds me of indicators in trading - they use candlesticks to predict changes in price, volume, etc. But they can't calculate market news, and all their assumptions will fail. So, it could be an ice cream day, and even if the correlation between ice cream sales and shark attacks is = 1, shark attacks will not increase on that day (probably), although ice cream sales will definitely increase.