# How could Bayesian neural networks be used for transfer learning?

In transfer learning, we use big data from similar tasks to learn the parameters of a neural network, and then fine-tune the neural network on our own task that has little data available for it. Here, we can think of the transfer learning step as learning a (proper) prior, and then fine-tuning as learning the posterior.

So, we can argue that Bayesian networks can also solve the problem of small data-set regimes. But, what are the directions that we can mix Bayesian neural networks with similar tasks to transfer learning, for example, few-shot learning?

They make sense when they both take a role as a solution to the low data regime problems, but I can't think of a mix of them to tackle this issue.

Is it possible, for example, to learn a BNN for which we have picked a good prior to learn the posterior with little data and use the weight distribution for learning our new task? Is there any benefit in this?

• I think you could interpret the way BNNs are formalised as a form of transfer learning, but this is an interpretation. So, you could train a BNN with some data $D$ to learn a posterior $p$ for the weights. Then you could retrain this BNN with new data $D'$ by starting from that posterior $p$ that you learned previously, i.e. we would start learning on the new task with the prior that corresponds to the previously learned posterior $p$. In this sense, we could call it transfer learning, but probably people would simply call it Bayesian inference, although the distributions have changed.
– nbro
Jul 7, 2021 at 10:39
• I am not aware of any research work that has attempted to tackle or interpret the transfer learning problem with BNNs, but it's also been a while since I worked with them.
– nbro
Jul 7, 2021 at 10:39