I am currently working with a categorical-binary RBM, where there are 50 categorical visible units and 25 binary hidden units. The categorical visible units are expressed in one-hot encoding format, such that if there is 5 categories, then the visible units are expressed as a $50 \times 5$ array, where each row is the one-hot encoding of a category from 1 to 5.

Ideally, the RBM should be able to reconstruct the visible units. However, since the visible units are in one-hot encoding, then the visible units array contains a lot of zeros. This means the RBM quickly learns to guess all zeros for the entire array to minimize the reconstruction loss. How can I force the RBM to not do this and to instead guess 1's where the category occurs and 0's otherwise?

Note that I would still have this problem with a regular autoencoder.

  • $\begingroup$ I don't think the problem you are experiencing is related to overfitting (as the title suggests), but due to sparsity of output in the target population for the generator. Perhaps you could edit the title? Interesting problem though $\endgroup$ Nov 10, 2020 at 15:48
  • $\begingroup$ Yes you are correct. Here is an article discussing this exact problem. However, this article discusses autoencoders, not RBMs. Got any ideas? $\endgroup$
    – mhdadk
    Nov 10, 2020 at 16:23


You must log in to answer this question.