I am currently working with a categorical-binary RBM, where there are 50 categorical visible units and 25 binary hidden units. The categorical visible units are expressed in one-hot encoding format, such that if there is 5 categories, then the visible units are expressed as a $50 \times 5$ array, where each row is the one-hot encoding of a category from 1 to 5.
Ideally, the RBM should be able to reconstruct the visible units. However, since the visible units are in one-hot encoding, then the visible units array contains a lot of zeros. This means the RBM quickly learns to guess all zeros for the entire array to minimize the reconstruction loss. How can I force the RBM to not do this and to instead guess 1's where the category occurs and 0's otherwise?
Note that I would still have this problem with a regular autoencoder.