2
$\begingroup$

In transfer learning, we use big data from similar tasks to learn the parameters of a neural network, and then fine-tune the neural network on our own task that has little data available for it. Here, we can think of the transfer learning step as learning a (proper) prior, and then fine-tuning as learning the posterior.

So, we can argue that Bayesian networks can also solve the problem of small data-set regimes. But, what are the directions that we can mix Bayesian neural networks with similar tasks to transfer learning, for example, few-shot learning?

They make sense when they both take a role as a solution to the low data regime problems, but I can't think of a mix of them to tackle this issue.

Is it possible, for example, to learn a BNN for which we have picked a good prior to learn the posterior with little data and use the weight distribution for learning our new task? Is there any benefit in this?

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ I think you could interpret the way BNNs are formalised as a form of transfer learning, but this is an interpretation. So, you could train a BNN with some data $D$ to learn a posterior $p$ for the weights. Then you could retrain this BNN with new data $D'$ by starting from that posterior $p$ that you learned previously, i.e. we would start learning on the new task with the prior that corresponds to the previously learned posterior $p$. In this sense, we could call it transfer learning, but probably people would simply call it Bayesian inference, although the distributions have changed. $\endgroup$
    – nbro
    Jul 7, 2021 at 10:39
  • 2
    $\begingroup$ I am not aware of any research work that has attempted to tackle or interpret the transfer learning problem with BNNs, but it's also been a while since I worked with them. $\endgroup$
    – nbro
    Jul 7, 2021 at 10:39

1 Answer 1

Reset to default
0
$\begingroup$

Well, I would say, that purpose of Bayesian inference is not transfer learning, but uncertainty estimation.

In case you have good feature extractor in the beginning, you can adjust small number of parameters, like few last layers to achieve good quality in few epochs.

However, this is about adjusting the means of distributions over each weight.

Concerning the variance, I think transfer learning is inapplicable since the source and target distributions can be very different. For example, ImageNet is a broad and diverse dataset with many classes, and the target problem can involve only a few classes. Most probably, uncertainty estimate and the standard deviations of model weights on the ImageNet would be larger, than for the model, trained solely on the target task.

Improve this answer
$\endgroup$
1
  • $\begingroup$ I do not get the argument of whether transfer learning would be make more or less sense for BNNs compared to usual ANNs. Whether the weights are distribution functions or pure numbers; what's the difference here? Don't BNNs overfit less, so transfer learning would actually make more sense? $\endgroup$
    – Mathy
    Jun 23, 2022 at 7:57

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .