2
$\begingroup$

I wonder if the following equation (you can find it in almost every ML book) refers to a general assumption that we make when using machine learning:

$$y = f(x)+\epsilon,$$

where $y$ is our output, $f$ is e.g. a neural network and $\epsilon$ is an independent noise term.

Does this mean that we assume the $y$'s contained in our training data set come from a noised version of our network output?

$\endgroup$
3
$\begingroup$

Not necessarily. The neural network (or whatever else you use) is a model of what you are trying to do, and usually models are not able to perfectly model reality, as it is too complex. A noise term is generally used to represent that, ie the imperfection of the model's relationship with the actual world.

$\endgroup$
1
$\begingroup$

That equation is just an assumption that we make about the relationship between a response variable (aka dependent variable) $y$ and a predictor (aka independent variable) $x$, i.e. the response variable (target) is an unknown function $f$ of the predictor $x$ plus some noise $\epsilon$ due to e.g. measurement errors (caused e.g. by damaged sensors). So, if you have a dataset $D = \{(y_i, x_i)\}_{i=1}^N$, you assume that $y_i = f(x_i) + \epsilon, \forall i$. The goal (in supervised learning) is then to estimate $f$ with e.g. a neural network $\hat{f}_\theta$, so the goal is to find a function $\hat{f}_\theta$ such that $\hat{f}_\theta(x_i) = y_i$, so, in practice, you often ignore $\epsilon$ because that is associated with irreducible errors.

You can find that equation on page 16 of the book An Introduction to Statistical Learning. There you will also find more info about the goal of (statistical) supervised learning and why $\epsilon$ is irreducible.

So, the answer to your question is no, given that $f$ there is not the neural network but an unknown function. If your neural network $\hat{f}$ was equal to $f$, then, yes, but, of course, in practice, this will almost never be the case.

$\endgroup$
2
  • $\begingroup$ thanks for answering! How does this apply to the Kalman Filter where we say that $y = Hx + r$, where r is the measurement noise. If there is no measurement noise, then the observations can be fully derived from the state? $\endgroup$
    – f_3464gh
    Mar 31 at 11:15
  • $\begingroup$ @m_3464gh I'm not currently familiar with the details of the Kalman filter, so I can't answer your question. I was about to suggest you to ask a separate question, but you have already done that here. $\endgroup$
    – nbro
    Mar 31 at 12:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.