cross_val_score of sklearn and LinearRegression scoring method

Question

The function cross_val_score uses the estimator’s default scorer (if available) and LinearRgression (the estimator I use) uses The coefficient of determination (which is defined as $R^2 = 1 - \frac{u}{v}$, where $u$ is the residual sum of squares ((y_true - y_pred)** 2).sum() and $v$ is the total sum of squares ((y_true - y_true.mean()) ** 2).sum(). The best possible score is 1.0 and it can be negative (because the model can be arbitrarily worse). A constant model that always predicts the expected value of y, disregarding the input features, would get a score of 0.0.)

Is y_true.mean the y_true.mean of the training set or the testing set? If it's the one of the testing set, isn't it "cheating" i.e. we compare our predictions to a method that has inferred something from the testing set?

So doing better than a baseline wouldn't be having $R^2 > 0$ but rather $R^2 > -0.01$ or something like this?

Answer 1

The cross_val_score function of the sci-kit-learn library uses the test dataset's mean to calculate the R2 score during each cross-validation fold. This is not called cheating since the R2 score is designed to evaluate how well the model is performing generalized on new unseen datasets and using the test set's mean is a standard part of the R² calculation.

you are right about it y_true.mean is from the testing dataset but it is not cheating because R2 > 0 means the model performs better than a constant baseline that predicts the mean of the test dataset.

If your model can't beat this simple baseline, it means the model isn't effectively learning the relationship between the input features and the target.

The baseline comparison is not meant to "cheat" by using information from the testing set; rather, it is a fair baseline that checks if your model is doing better than simply predicting the average of the testing set’s target values.

Should R² > -0.01 Be the Baseline?

Not really. The baseline you are competing against is R² = 0, which is the score of a constant model that predicts the mean of y_test. A score below 0 means the model is performing worse than simply predicting the mean of the test data, but R² > 0 is the standard for doing better than a baseline model.

I am not going much deep into the interpretation of the r2 score like what the value of r2 suggests about the model results since Luca Anzalone has explained well in above answer

for the question So doing better than a baseline wouldn't be having R2>0 but rather R2>−0.01 or something like this?

=> No, doing better than the baseline is indicated by having an R² score greater than 0. it is due to following reasons

An R² score of 0 means that your model performs as well as a constant model that predicts the mean of the target values from the test set, ignoring the input features.

An R² score greater than 0 means your model is doing better than the baseline (constant mean prediction).

An R² score less than 0 means your model is doing worse than the baseline.

If your model had an R² score of something like -0.01, that would imply that the model's predictions are slightly worse than just predicting the mean of the test set. But the true goal is to have R² > 0, which indicates that your model is leveraging the features to make predictions that outperform the simple mean baseline.

if you have still doubt feel free to ask and also ignore some notation mistakes since i am new at typing notation on stack :)

Answer 2

Let's clarify this with a numerical example. Let's assume y_true = [0, 1, 1] to be the true class labels of $N=3$ test points. Therefore, y_true.mean() = 2/3.

Case 1: everthing is wrongly predicted, so y_pred = [1, 0, 0]. Let's compute $u$ and $v$. $$ \begin{align} u &= \sum(y_\text{true}-y_\text{pred})^2 = (0-1)^2+(1-0)^2 +(1-0)^2 = 3 \\ v &= \sum \Big(y_\text{true}-\frac23\Big)^2 = (-2/3)^2+(1/3)^2+(1/3)^2 = \frac69 = \frac23 \end{align}$$ Now we compute $R^2$, so: $$R^2 = 1 - \frac{u}{v} = 1 - \frac{3}{2/3} = 1 - 2 = \mathbf{-1}$$ You get -1 because y_pred is always wrong. so you can think of $u$ as counting the number of errors (it's so if you predict the label to be exactly zero or one), whereas $v$ is the ratio of positive labels.

Case 2: perfect predictions, so y_pred = y_true. $$ \begin{align} u &= (0-0)^2+(1-1)^2 +(1-1)^2 = 0\quad\quad\text{(no errors)} \\ R^2 &= 1 -\frac{0}{2/3} = 1 \end{align}$$ $R^2=1$ because every label is correctly predicted, so there are no errors and the correlation is the most possible.

Case 3: some errors, say y_pred=[0, 1, 0]. Again: $$ \begin{align} u &= (0-0)^2+(1-1)^2 +(1-0)^2 = 1 \\ R^2 &= 1 -\frac{1}{2/3} = 1 - \frac32 = -\frac12 \end{align}$$ $R^2=-0.5$, so negative correlation. From this it seems that $R^2$, in general, gives more importance to correctly predicting the positive class.

So, there is no issue about using the mean of the true labels because that info is not used during training but only to compute an evaluation metric.