0
$\begingroup$

In artificial intelligence, especially in machine learning, the inputs and outputs of neurons in a neural network can be viewed as random variables.

And this view is highly useful in many ways. The usefulness of this view manifests in applying various mathematical results of random variables in understanding outputs and other aspects of neural networks. Other aspects include loss functions also.

I want to know whether it is useful to view weights of neural networks as random variables? What does the literature say about this view?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

In Bayesian statistics, as opposed to frequentist statistics, you can model the parameters as random variables. Bayesian machine learning is the application of Bayesian statistics in the context of machine learning. The specific application of Bayesian statistics to learning in neural networks is denoted as Bayesian deep learning.

Even just in the simple example of maximum a posteriori (MAP) estimation, you model the weights as random variables. For example, you can view weight decay in neural networks as MAP estimation when you put a prior on the weights, i.e. you assume that the weights are random variables or samples from a distribution. Here you can find more details about MAP estimation and how it's different from MLE.

In MAP, we only find a point estimate of the weights. However, we could also learn a probability distribution over the weights. If you do this with neural networks, we have Bayesian neural networks. Two of the first people that worked on this were David J. C. MacKay and Radford M. Neal, but only very recently did Bayesian neural networks really become popular, with Alex Graves and Blundel et al. Anyway, there are different approaches to Bayesian learning in neural networks. There are techniques based on Monte Carlo methods (e.g. Neal) or variational inference (e.g. Graves and Blundell).

So, what are the advantages of viewing weights as random variables? Well, you can derive MAP and Bayesian neural networks. Basically, all the advantages of Bayesian statistics.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Nice answer, thanks! Do you have any practical examples of where BNNs are used nowadays, maybe even more successful than other alternatives? $\endgroup$
    – Mathy
    Commented Jun 23, 2022 at 8:44
  • 1
    $\begingroup$ @Mathy Honestly, no. It's been a while since I worked with them, so I forgot a few things. Maybe that could be a good question for our site, so I recommend that you ask it ;) Make sure to you use the tags state-of-the-art and reference-request and/or to provide some context (e.g. a brief description of BNNs). $\endgroup$
    – nbro
    Commented Jun 23, 2022 at 13:53
  • $\begingroup$ Thank you for the idea; just done so ;-) ai.stackexchange.com/questions/36040/… $\endgroup$
    – Mathy
    Commented Jun 25, 2022 at 8:29

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .