In supervised machine learning, it is common to say that we learn a function of the form

$$y=g(x) + \epsilon.$$

Generally, $\epsilon$ is used to denote noise or, more precisely, any influence by latent variables such as measurement inaccuracies (right?).

Is it, therefore, correct to say that we use $\epsilon$ to denote the model's imperfection to the real world (caused by anything unknown)?


1 Answer 1


Yes, precisely.

What do you mention is known in the literature as the Bayes Error. See page top of the page 114 https://www.deeplearningbook.org/contents/ml.html.

The ideal model is an oracle that simply knows the true probability distribution that generates the data. Even such a model will still incur some error on many problems, because there may still be some noise in the distribution. In the case of supervised learning, the mapping from $\mathbf{x}$ to $y$ may be inherently stochastic, or $y$ may be a deterministic function that involves other variables besides those included in $\mathbf{x}$. The error incurred by an oracle making predictions from the true distribution $p(\mathbf{x}, y)$ is called the Bayes error.

Regardless of how clever is your model, the best error you can achieve for the prediction on the data distribution is $\varepsilon$. Note that it holds for the whole data distribution, not a sample of data.

Say, you would like to fit something like $\sin(x) + \varepsilon$. There are 10 points, and one can fit them perfectly with the 9th-degree polynomial, but this an error on training data, and, in case one samples more data points, the error will be likely do exceed the optimal $\varepsilon$.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .