In transfer learning, we use big data from similar tasks to learn the parameters of a neural network, and then fine-tune the neural network on our own task that has little data available for it. Here, we can think of the transfer learning step as learning a (proper) prior, and then fine-tuning as learning the posterior.
So, we can argue that Bayesian networks can also solve the problem of small data-set regimes. But, what are the directions that we can mix Bayesian neural networks with similar tasks to transfer learning, for example, few-shot learning?
They make sense when they both take a role as a solution to the low data regime problems, but I can't think of a mix of them to tackle this issue.
Is it possible, for example, to learn a BNN for which we have picked a good prior to learn the posterior with little data and use the weight distribution for learning our new task? Is there any benefit in this?