Artificial Intelligence Stack Exchange is a question and answer site for people interested in conceptual questions about life and challenges in a world where "cognitive" functions can be mimicked in purely digital environment. It only takes a minute to sign up.
Sign up to join this community
I wonder if the following equation (you can find it in almost every ML book) refers to a general assumption that we make when using machine learning:
$$y = f(x)+\epsilon,$$
where $y$ is our output, $f$ is e.g. a neural network and $\epsilon$ is an independent noise term.
Does this mean that we assume the $y$'s contained in our training data set come from a noised version of our network output?
Not necessarily. The neural network (or whatever else you use) is a model of what you are trying to do, and usually models are not able to perfectly model reality, as it is too complex. A noise term is generally used to represent that, ie the imperfection of the model's relationship with the actual world.
That equation is just an assumption that we make about the relationship between a response variable (aka dependent variable) $y$ and a predictor (aka independent variable) $x$, i.e. the response variable (target) is an unknown function $f$ of the predictor $x$ plus some noise $\epsilon$ due to e.g. measurement errors (caused e.g. by damaged sensors). So, if you have a dataset $D = \{(y_i, x_i)\}_{i=1}^N$, you assume that $y_i = f(x_i) + \epsilon, \forall i$. The goal (in supervised learning) is then to estimate $f$ with e.g. a neural network $\hat{f}_\theta$, so the goal is to find a function $\hat{f}_\theta$ such that $\hat{f}_\theta(x_i) = y_i$, so, in practice, you often ignore $\epsilon$ because that is associated with irreducible errors.
You can find that equation on page 16 of the book An Introduction to Statistical Learning. There you will also find more info about the goal of (statistical) supervised learning and why $\epsilon$ is irreducible.
So, the answer to your question is no, given that $f$ there is not the neural network but an unknown function. If your neural network $\hat{f}$ was equal to $f$, then, yes, but, of course, in practice, this will almost never be the case.
