What are the 'noisy factors' leading to overfitting?

Question

Consider the following excerpt from section 5.5 Regularization (p. 13) of this chapter Logistic Regression.

There is a problem with learning weights that make the model perfectly match the training data. If a feature is perfectly predictive of the outcome because it happens to only occur in one class, it will be assigned a very high weight. The weights for features will attempt to perfectly fit details of the training set, in fact too perfectly, modeling noisy factors that just accidentally correlate with the class. This problem is called overfitting.

What are the 'noisy factors' here? Does it refer to the features that are irrelevant to the class label?

Or does it mean the noise/errors in the values taken by features that accidentally correlate with the class label?

Answer 1

Please note:

• I am only referring the decision boundary to be a line for simplicity, more often than not it is a hyperplane which is difficult to visualize and spans over n dimensions where n is the dimensionality of your feature space.
• The explanation is toned in a more general way for emphasizing explainability.

Answer

What are the 'noisy factors' here? Does it refers to the features that are irrelevant to the class label?

• Not Necessarily.

Or does it mean the noise/errors in the values taken by features that accidentally correlate with the class label?

• I'm not quite sure I understand.

However

Noisy factors as the literature puts it are outliers in a finite data class. Imagine a dataset where we are asked to calculate the average of a set of numbers. Lets say the numbers are the set S = {2,2,2,2,2,2,2,1000}. The mean value in the case mentioned is 2 if it wasn't for the 1000 at the end.

• 1000 could be an outlier when you are trying to approximate the mean of the set with some algorithm. The algorithm is more likely to encounter a 2 in the unseen test set than the 1000 it encountered once in a training set.
• When an algorithm like Linear Regressions "learns" something it is actually learning the weights and intercepts which modify the position of the line in the decision space.
• The modifications are Translation(Additive operations) and Rotation(Multiplicative operations) to the said line. It is commonly referred to as the "weights" and "biases" respectively in case of a simple line Y = mx + c where m is the slope and c is the intercept.
• The idea behind noise in the above text is that this: When an algorithm is trying to "learn" these weights you would want it to ignore the outliers - "Generalize" but you also want it to not ignore the feature data completely and introduce randomness - "Specialization".
• How well the line is placed in your feature space is what determines your Classification Effectiveness.
• In an ideal world you would want your decision boundary to ignore such outliers which are called noise in the above literature.

Pictures (because everyone likes them)

• In the picture below the left image is what a good decision boundary is and the right image is what an overfitted Decision boundary is.
• In the left-image case you are trading misclassifications at the cost of better generalizability(Consider that more likely you are going to see an x in the 2nd quadrant and the o was maybe an outlier). You do not fit your training set completely with a 100% accuracy.

TL;DR

You want the model to learn that the x(s) are on the right and the o(s) are on the left but not so specifically as to which x was where. The x in the 4th quadrant and the o in the second quadrant are noisy factors.