In the deep learning course I took at the university, the professor touched upon the subject of the Restricted Boltzmann Machine. What I understand from this subject is that this system works completely like Artificial Neural Networks. During the lecture, I asked the Professor the difference between these two systems, but I still did not fully understand. In general, there is an input layer and a hidden layer in both, and the weights are updated with forward-backward propagation. Can someone explain the exact difference between them?
$\begingroup$
$\endgroup$
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
A Boltzmann Machine is a probabilistic graphical model which follows Boltzmann distribution:
$$p(v,h) = \frac{e^{-E(v,h)}}{\sum_{v,h} e^{-E(v,h)}}$$ where $E(v,h)$ is known as the energy function.
An RBM is a Boltzmann machine with a restriction that there are no connections between any two visible nodes or any two hidden nodes, which gives it the structure similar to a 2-layer Artificial Neural Network.
The difference is that RBM, being an unsupervised model, is trained by minimizing the energy function. While an artificial neural network can have many hidden layers along with an output layer, and is trained by optimizing the loss between the values of output layer and the values of target variable.
A special case of RBM which has binary visible nodes and binary hidden nodes, also known as Bernoulli RBM has an API available in scikit-learn. They have also documented the learning algorithm here. In this example, they show how Bernoulli RBM can be used to perform effective non-linear feature extraction which can be fed to the LogisticRegression classifier for digit classification.
$\begingroup$
$\endgroup$
You can find in this paper that RBM is a specific type of artificial neural networks. Hence, the term Artificial Neural Network is more general than RBF.
It is an important property that single as well as stacked RBMs can be reinterpreted as deterministic feed-forward neural networks. Than they are used as functions from the domain of the observations to the expectations of the latent variables in the top layer. Such a function maps the observations to learnt features, which can, for example, serve as input to a supervised learning system. Further, the neural network corresponding to a trained RBM or DBN can be augmented by an output layer, where units in the new added output layer represent labels corresponding to observations. Then the model corresponds to a standard neural network for classification or regression that can be further strained by standard supervised learning algorithms. It has been argued that this initialization (or unsupervised pretraining) of the feed-forward neural network weights based on a generative model helps to overcome problems observed when training multi-layer neural networks.
Of course, we have other types of neural networks like Recurrent Neural Networks expressing them as an RBF is not straightforward.
