Artificial Intelligence Stack Exchange is a question and answer site for people interested in conceptual questions about life and challenges in a world where "cognitive" functions can be mimicked in purely digital environment. It only takes a minute to sign up.
Sign up to join this community
What are the main differences and similarities between sparse autoencoders and convolution autoencoders?
When should one be preferred over the other? What are their applications?
(References are welcome. Somehow I was not able to find any comparisons of these autoencoders although I looked in a few textbooks and searched for material online. I was able to find the descriptions of each autoencoder separately, but what I am interested in is the comparison.)
Sparse auto-encoders (SAEs) are auto-encoders that impose constraints on the parameters so that they are sparse (i.e. zero or close to zero). This can be achieved in different ways. For example, you can train an auto-encoder with a loss function that includes a penalty term (to constraint the parameters to be close to zero or zero) or you e.g. set the smallest activations to zero.
Convolution auto-encoders (CAEs) are auto-encoders that use the convolution operation. So, they can be viewed as the auto-encoder version of convolutional neural networks. For this reason, they are particularly suited to compress and reconstruct images. The authors of the original paper, Stacked Convolutional Auto-Encoders for Hierarchical Feature Extraction, train them with gradient descent and back-propagation by minimizing the mean squared error, so there's e.g. no penalty term, but you can probably combine SAEs with CAEs.
Of course, you could say that CAEs are sparse with respect to the traditional auto-encoder (in the same way that you can say that CNNs are sparse with respect to fully connected neural networks), so, in this sense, CAEs are also sparse auto-encoders.
