What is the sensitivity of coefficients in multicollinearity?

Question

I've heard that the coefficients in multicollinearity are very sensitive, and can change due to small changes in the data.... Isn't it a problem with the dataset itself that we have different data? And can you give me an (realistic) example where the coefficients directly change very much from small changes in the data?

Answer 1

You can start to see this in a fairly straightforward simulation using a linear regression. Simulate 1000 linear regressions with highly correlated features and 1000 with uncorrelated features. Which one has a wider range of fitted coefficients? How does that range depend on the extent of the correlation?

The underlying math is that the multicollinearity inflates the coefficient standard errors through the so-called variance inflation factor.

I don’t have access to R until tomorrow night, but something like the code below should do the trick.

library(MASS)
set.seed(2024)
N <- 100
R <- 1000
b_corr <- b_ind <- rep(NA, R)
for (i in 1:R){

X_corr <- MASS::mvrnorm(N, c(0, 0), matrix(c(1, 0.99, 0.99, 1), 2, 2)
X_ind <- cbind(X_corr[, 1], sample(X_corr[, 2], N, replace = F)

e <- rnorm(N)
y_corr <- X_corr %*% c(1, 1) + e
y_ind <- X_ind %*% c(1, 1) + 

L_corr <- lm(y_corr ~ X_corr)
L_ind <- lm(y_ind ~ X_ind)

b_corr[i] <- coef(L_corr)[2]
b_ind[i] <- coef(L_ind)[2]
}
var(b_corr)
var(b_ind)

(Without access to R, I can’t guarantee that the above code will compile, but it gives the gist of how to simulate this.)

How much larger is the variance with highly correlated features (one form of feature multicollinearity) vs the variance when the features are independent yet everything else is the same?

If you want to see this from real data, perhaps try taking various subsets from a dataset with feature multicollinearity and watch how the coefficients change as the make changes to the training data.

Answer 2

This isn't typically due to a flaw in the dataset but rather reflects the structural overlap in predictor information within the data. When predictors are nearly collinear the matrix of predictor values becomes nearly singular, thus it becomes difficult to pinpoint unique contributions of each predictor. During estimation the linear regression procedure seeks a best fit but struggles to allocate weight accurately between highly correlated variables. As a result small changes in data with limited samples could possibly produce large shifts in coefficients as the model attempts to settle on new estimates.

Here's a concrete example written in python.

import pandas as pd
from sklearn.linear_model import LinearRegression

# Initial data setup
data = {
    'SquareFootage': [1500, 2000, 2500, 3000],
    'Rooms': [5, 7, 8, 9],
    'Price': [200000, 280000, 340000, 400000]
}

# Create DataFrame
df = pd.DataFrame(data)

# Function to fit model and print coefficients
def fit_model(dataframe):
    X = dataframe[['SquareFootage', 'Rooms']]
    y = dataframe['Price']
    model = LinearRegression()
    model.fit(X, y)
    return model.coef_

# Initial fit with original data
print("Original Coefficients:")
original_coeffs = fit_model(df)
print(f"Square Footage: {original_coeffs[0]:.2f}, Rooms: {original_coeffs[1]:.2f}")

# Make a small change to the data (decrease Square Footage for House C a bit)
df_modified = df.copy()
df_modified.loc[2, 'SquareFootage'] = 2400

# Fit model with modified data
print("\nModified Coefficients (after small data change):")
modified_coeffs = fit_model(df_modified)
print(f"Square Footage: {modified_coeffs[0]:.2f}, Rooms: {modified_coeffs[1]:.2f}")

After running the program you get below results for comparison where the weights are sensitive :

Original Coefficients:
Square Footage: 80.00, Rooms: 20000.00

Modified Coefficients (after small data change):
Square Footage: 65.98, Rooms: 25773.20