2
$\begingroup$

I'm using a Restricted Boltzmann Machine (RBM) as an autoencoder. For now, I use a simple architecture of two layers, the input (~100 nodes) and the output (3 nodes) layers. I'm thinking to add more hidden layers.

Are there some improvements in encoding by adding multiple hidden layers? If yes, how can multiple layers improve the encoding?

$\endgroup$
  • 1
    $\begingroup$ It's been too long since I've looked at them for me to feel comfortable writing a proper answer... but you're going to want to look into "Deep Belief Networks" $\endgroup$ – Dennis Soemers Jun 23 at 9:29
1
$\begingroup$

First of all, when you add hidden layers, or stack RBMs, you get a Deep Belief Network (DBN). Your question then deals with the comparison of DBNs and RBMs.

There are some elements to answer this question in the article Representational Power of Restricted Boltzmann Machines and Deep Belief Networks by Nicolas Le Roux, which can be found summarized in these course slides. The main results are:

Restricted Boltzmann Machines:

  • Increasing the number of hidden units improves representational ability.

  • With an unbounded number of units, any distribution over $\{0,1\}^n$ can be approximated arbitrarily well.

Deep Belief Networks:

  • Adding additional layers using greedy contrastive divergence training does not provide additional benefit.

  • There remain open questions about the benefits additional layers add.

I emphasized the $4^{th}$ point that covers the most your question. It does not mean that additional layers are useless, but since RBM are universal approximators ($2^{nd}$ point), the benefits of adding layers seem less straightforward. They seem dependent on the first layer, the training procedure... You can see the article for more details and the open questions raised about DBNs. Note that this answer is an entry point on the topic, there might be more recent results following this article which I'm not familiar with...

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.