Does the encoding of a restricted Boltzmann machine improve with more layers?

Question

I'm using a restricted Boltzmann machine (RBM) as an autoencoder. For now, I use a simple architecture of two layers, the input (~100 nodes) and the output (3 nodes) layers. I'm thinking to add more hidden layers.

Are there some improvements in encoding by adding multiple hidden layers? If yes, how can multiple layers improve the encoding?

Answer 1

First of all, when you add hidden layers, or stack RBMs, you get a Deep Belief Network (DBN). Your question then deals with the comparison of DBNs and RBMs.

There are some elements to answer this question in the article Representational Power of Restricted Boltzmann Machines and Deep Belief Networks by Nicolas Le Roux, which can be found summarized in these course slides. The main results are:

Restricted Boltzmann Machines:

• Increasing the number of hidden units improves representational ability.

• With an unbounded number of units, any distribution over $\{0,1\}^n$ can be approximated arbitrarily well.

Deep Belief Networks:

• Adding additional layers using greedy contrastive divergence training does not provide additional benefit.

• There remain open questions about the benefits additional layers add.

I emphasized the $4^{th}$ point that covers the most your question. It does not mean that additional layers are useless, but since RBM are universal approximators ($2^{nd}$ point), the benefits of adding layers seem less straightforward. They seem dependent on the first layer, the training procedure... You can see the article for more details and the open questions raised about DBNs. Note that this answer is an entry point on the topic, there might be more recent results following this article which I'm not familiar with...