11
votes
Accepted
Loss jumps abruptly when I decay the learning rate with Adam optimizer in PyTorch
I see no reason why decaying learning rates should create the kinds of jumps in losses that you are observing. It should "slow down" how quickly you "move", which in the case of a loss that otherwise ...
- 9,794
9
votes
Accepted
Is plain autoencoder a generative model?
An autoencoder is not considered a generative model, because it only reconstructs the given input. You could use the decoder like a generative model by putting in different vectors. However, the ...
- 1,317
8
votes
Accepted
What is the difference between encoders and auto-encoders?
Theory
Encoder
In general, an Encoder is a mapping $f : X \rightarrow Y $ with $X$ Input Space and $Y$ Code Space
In case of Neural Networks, it is a Generative Model hence a function which is able ...
- 799
7
votes
Why doesn't VAE suffer mode collapse?
With Generative Adversarial Networks, all the generator cares about is fooling the discriminator. There's no requirement to be clever, or exhaustive, or make efficient use of the input space. As long ...
- 278
6
votes
Is it possible for a neural network to be used to compress data?
The auto-encoder (AE) can be used to learn a compressed representation (a vectorised hash value) of each observation in the training dataset, $z$, which can then be used to later retrieve the original ...
- 37k
6
votes
What are the purposes of autoencoders?
It is important to think about what sort of patterns in the data are being represented.
Suppose that you have a dataset of greyscale images, such that every image is a uniform intensity. As a human ...
- 169
6
votes
Accepted
What is an appropriate size for a latent space of (variational) autoencoders and how it varies with the features of the images?
You are asking about several things here and while related, solving one, will not necessarily "solve" your problem. Let's look at them separately:
Optimal dimension of the latent space.
...
- 176
4
votes
Why is the variational auto-encoder's output blurred, while GANs output is crisp and has sharp edges?
The key is: VAE usually use a small latent dimension, the information of input is so hard to pass through this bottleneck, meanwhile it tries to minimize the loss with the batch of input data, you ...
- 49
4
votes
Accepted
Concrete example of latent variables and observables plugged into the Bayes' rule
Let's assume the probability distributions are Gaussian (or normal) distributions. In other words, in the Bayes' rule
\begin{align}
p(z|x)=\frac{p(x|z)p(z)}{p(x)}
\tag{1}\label{1}
\end{align}
The ...
- 37k
4
votes
Why don't we use auto-encoders instead of GANs?
In fact, autoencoders are used for generative tasks. Have a look at Tutorial on Variational Autoencoders (VAEs).
The coolest thing about VAE is that abstract features can be easily amplified or ...
4
votes
Accepted
Why don't we use auto-encoders instead of GANs?
Auto-encoders are widely used and maybe even more used than GANs (in fact, auto-encoders are older than GANs, although the main general idea behind GANs is quite old). For example, auto-encoders are ...
- 37k
4
votes
Strange artifacts in autoencoder outputs
That's a classic checkerboard artifact. I would guess you're using a CNN as encoder/decoder architectures, since is well known that convolution layers, especially at the upsampling phase, cause these ...
- 4,753
3
votes
What are the purposes of autoencoders?
A use case of autoencoders (in particular, of the decoder or generative model of the autoencoder) is to denoise the input. This type of autoencoders, called denoising autoencoders, take a partially ...
- 37k
3
votes
How should we choose the dimensions of the encoding layer in auto-encoders?
The number of dimensions is a hyperparameter of your model, and you should do a hyperparameter search, like with any other parameters. There's also a tradeoff between dimension and training speed, so ...
3
votes
Autoencoder produces repeated artifacts after convergence
Perhaps you are getting checkerboard artifacts Explained here, solutions involve changing the kernel and stride size to prevent them from being not divisible. Besides that, a solution could be to ...
3
votes
Accepted
How does replacing states with latent representations help RL agents?
In short, it is much easier for the agent to learn from a smaller dimensional state space. This is because the agent must also do representation learning; i.e. it must also infer what the state is ...
- 4,400
3
votes
Accepted
How to determine the quality of synthetic data?
Due to subjective nature, quantitative evaluation of synthetic images is difficult in general. However, there are metrics like Inception Score or FID score that are used for evaluation of generative ...
- 238
2
votes
Can autoencoders be used for supervised learning?
One such paper I know of and which I implemented is Semi-Supervised Learning using Ladder Networks . I quote here their description of the model:
Our approach follows Valpola (2015), who proposed a ...
- 263
2
votes
Accepted
What are good parameters of an encoder?
I can not seem to grasp which specs make an encoder better than another one
In general, in unsupervised settings, we want to learn the probability distribution of the data p(x) by some latent ...
- 364
2
votes
Accepted
Do Le et al. (2012) train all three autoencoder layers at a time, or just one?
The paper refers to layers and sub-layers, and clearly indicates that one layer includes all three sub-layers, so when they say they train all three layers simultaneously, they are talking about the ...
- 151
2
votes
Why isn't the Credit Card Fraud Detection dataset from Kaggle already balanced?
I believe that the idea is to have a similar ratio of fraud/"normal transaction" as to the ones that bank encounter on real life.
If you balance it you will probably have a lot of false positive once ...
- 424
2
votes
What are the purposes of autoencoders?
PCA is a linear method that creates a transformation that is capable of changing the vectors projections (changing axis)
Since PCA looks for the direction of maximum variance it usually have high ...
2
votes
Accepted
How should I detect an object in a camera image?
Is this a good approach? Will I have a lot of trouble with different backgrounds?
A lot will depend on the nature of the backgrounds you have, and how well they encode/decode by themselves without ...
- 26.5k
2
votes
If I use MobileNetV2 for the encoder, can I use a different architecture for the decoder?
Other replies are commenting on the skip connections for a U-Net. I believe you want to exclude these skip connections from your auto-encoder. You say you want to use the auto-encoder for unsupervised ...
- 21
2
votes
Using ML to encypher data for production
It sounds like you are trying to compress data, and then recover the same data later.
The most common tool for this task is an autoencoder. This model accepts data as input, and then learns to ...
- 9,037
2
votes
How are small scale features represented in an Inverse Graphics Network (autoencoder)?
Simply said, there is no specific "meaning" to the features generated. They are simply features that are fitted through math and calculus, and nobody knows what they represent exactly, and will never ...
- 1,725
2
votes
Accepted
Why is exp used in encoder of VAE instead of using the value of standard deviation alone?
In the source code, the author defines sd by
sd = 0.5 * tf.layers.dense(x, units=n_latent)
which means that $\...
- 261
2
votes
In variational autoencoders, why do people use MSE for the loss?
On page 5 of the VAE paper, it's clearly stated
We let $p_{\boldsymbol{\theta}}(\mathbf{x} \mid \mathbf{z})$ be a multivariate Gaussian (in case of real-valued data) or Bernoulli (in case of binary ...
- 37k
2
votes
Accepted
In variational autoencoders, why do people use MSE for the loss?
If $p(x|z) \sim \mathcal{N}(f(z), I)$, then
\begin{align}
\log\ p(x|z)
&\sim \log\ \exp(-(x-f(z))^2) \\
&\sim -(x-f(z))^2 \\
&= -(x-\hat{x})^2,
\end{align}
where $\hat{x}$, the ...
- 149
Only top scored, non community-wiki answers of a minimum length are eligible