11

This is a very important problem that is usually overlooked. In fact, when training a neural network, there's often the implicit assumption that the data is independent and identically distributed, i.e., you do not expect the data to come from a distribution different than the distribution from which your training data comes, so there's also the implicit ...


3

Your description of what is going on is more or less correct, although I am not completely sure that you have really understood it, given your last question. So, let me enumerate the steps. The computation of the posterior is often intractable (given that the evidence, i.e. the denominator of the right-hand side of the Bayes' rule, might be numerically ...


2

Yes, there is some research on this topic. It's often called Bayesian machine learning or Bayesian deep learning (but I don't think this is a good name because there are models that aren't really based on a direct application of Bayesian statistics). Some ML/DL models that provide some kind of uncertainty estimation are, for example, Monte Carlo Dropout (MC ...


1

We want a distribution over $w$, don't we? Yes. You want to obtain a distribution over the parameters, which models the uncertainty about the parameters. This distribution over the parameters can induce a probability distribution over the possible functions consistent with your data. Why is $a$ integrated out here and not $w$? This is just the definition ...


1

The likelihood depends on the task that you are solving, so this is similar to traditional neural networks (in fact, even these neural networks have a probabilistic/Bayesian interpretation!). For binary classification, you should probably use a Bernoulli, which, in practice, corresponds to using a sigmoid with a binary cross-entropy (you can show that the ...


1

Your intuition is right. The main reason why a deterministic function can be undesirable (or even dangerous, as I will explain below with an example) is that we may not have enough data to learn the correct function, so we may end up learning the incorrect one. Right now, no other reason, from a theoretical point of view, comes to my mind, but below I will ...


1

In expectation step, firstly we calculate the posterior of latent variable $Z$ and then the $Q(θ | θ^{(t)})$ is defined as the expected value of the log likelihood of $θ$, with respect to the current conditional contribution of $Z$ given $X$ and the current estimates of $θ^{(t)}$. In maximization step, we update $θ$ using the argmax on $Q$, with respect to $...


Only top voted, non community-wiki answers of a minimum length are eligible