I am trying to assess an encoder in my autoencoder. I can not seem to grasp which specs make an encoder better than other one in, lets say, unsupervised learning. For example, I am trying to teach my neural network to classify cats, so that when I provide a picture of a bird, my autoencoder would tell me that it is not a picture of a cat. I am trying to understand what exact specs make my encoder (and decoder) better? I understand it is all about chosen weights but is it possible to be more specific?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
7
I can not seem to grasp which specs make an encoder better than another one
In general, in unsupervised settings, we want to learn the probability distribution of the data p(x) by some latent variables that explain the variations observed in the training set.
The autoencoder family (Variational, Denoising, Contrastive, Sparse) try to approximate p(x) so we have a performance metric to tell us how our model is doing. e.g. (negative log likelihood of p(x))
lets say, unsupervised learning. For example, I am trying to teach my neural network to classify cats,
If you use some autoencoder model to learn the distribution of cats, you could use the encoder part to further augmented with a linear classifier to discriminate between cats and other categories Therefore you have an intrinsic task (learn a good representation of the data distribution) and an extrinsic task (learn to classify cat vs not a cat). So you could do a hyper-parameter search for the model that best suits your problem by measuring its accuracy on the extrinsic task.
Side note:
GAN (Generative Adversarial Network) is a generative model, it provides some way of interacting less directly with this p(x) by drawing samples from it starting without any input. thus the situation here is different.
-
$\begingroup$ the link mentioned above is simply explaining the naive autoencoder, the bottom line, data in high dimensional space always lie in a lower dimensional space called manifold, where the probability concentrates and decrease abruptly as you go far away from the manifold, of course, the manifold could be composed of a set of non-connected manifolds (modes), for more explanation see ch14 of deeplearningbook.org $\endgroup$ Commented May 21, 2018 at 7:28
-
$\begingroup$ So if I understood correctly, for an autoencoder, Id calculate the probability of a function h given data x that best describes x? Lets say, my function that describes x is x^3 + 1, so that means I would go through each vector of x and calculate its probability falling under that function? $\endgroup$– GabrieleCommented May 21, 2018 at 8:20
-
1$\begingroup$ If the data distribution is concentrated on x^3+1 with some variance, the model will try to put more probability on points that are similar to the training data and puts a lower probability on points that don't resemble the data distribution based on the maximum likelihood principle. $\endgroup$ Commented May 21, 2018 at 8:28
-
$\begingroup$ What if I do not know the probability? What if I start with a function x +1 even though the real h is x^3 + 1 $\endgroup$– GabrieleCommented May 21, 2018 at 8:42
-
1$\begingroup$ You don't need to know the probability, you just have training data, where they have some internal structure, you just exploit it by letting the model learns P(x) $\endgroup$ Commented May 21, 2018 at 8:45