10 votes
Accepted

Loss jumps abruptly when I decay the learning rate with Adam optimizer in PyTorch

I see no reason why decaying learning rates should create the kinds of jumps in losses that you are observing. It should "slow down" how quickly you "move", which in the case of a loss that otherwise ...
user avatar
  • 9,379
7 votes
Accepted

What is the difference between encoders and auto-encoders?

Theory Encoder In general, an Encoder is a mapping $f : X \rightarrow Y $ with $X$ Input Space and $Y$ Code Space In case of Neural Networks, it is a Generative Model hence a function which is able ...
user avatar
6 votes

Is it possible for a neural network to be used to compress data?

The auto-encoder (AE) can be used to learn a compressed representation (a vectorised hash value) of each observation in the training dataset, $z$, which can then be used to later retrieve the original ...
user avatar
  • 33.8k
6 votes

What are the purposes of autoencoders?

It is important to think about what sort of patterns in the data are being represented. Suppose that you have a dataset of greyscale images, such that every image is a uniform intensity. As a human ...
user avatar
  • 161
4 votes

Why is the variational auto-encoder's output blurred, while GANs output is crisp and has sharp edges?

The key is: VAE usually use a small latent dimension, the information of input is so hard to pass through this bottleneck, meanwhile it tries to minimize the loss with the batch of input data, you ...
user avatar
4 votes
Accepted

Concrete example of latent variables and observables plugged into the Bayes' rule

Let's assume the probability distributions are Gaussian (or normal) distributions. In other words, in the Bayes' rule \begin{align} p(z|x)=\frac{p(x|z)p(z)}{p(x)} \tag{1}\label{1} \end{align} The ...
user avatar
  • 33.8k
4 votes

Why don't we use auto-encoders instead of GANs?

In fact, autoencoders are used for generative tasks. Have a look at Tutorial on Variational Autoencoders (VAEs). The coolest thing about VAE is that abstract features can be easily amplified or ...
user avatar
4 votes
Accepted

Why don't we use auto-encoders instead of GANs?

Auto-encoders are widely used and maybe even more used than GANs (in fact, auto-encoders are older than GANs, although the main general idea behind GANs is quite old). For example, auto-encoders are ...
user avatar
  • 33.8k
4 votes

Is plain autoencoder a generative model?

An autoencoder is not considered a generative model, because it only reconstructs the given input. You could use the decoder like a generative model by putting in different vectors. However, the ...
user avatar
  • 356
3 votes

Why doesn't VAE suffer mode collapse?

With Generative Adversarial Networks, all the generator cares about is fooling the discriminator. There's no requirement to be clever, or exhaustive, or make efficient use of the input space. As long ...
user avatar
  • 228
3 votes

What are the purposes of autoencoders?

A use case of autoencoders (in particular, of the decoder or generative model of the autoencoder) is to denoise the input. This type of autoencoders, called denoising autoencoders, take a partially ...
user avatar
  • 33.8k
3 votes

How should we choose the dimensions of the encoding layer in auto-encoders?

The number of dimensions is a hyperparameter of your model, and you should do a hyperparameter search, like with any other parameters. There's also a tradeoff between dimension and training speed, so ...
user avatar
3 votes
Accepted

How does replacing states with latent representations help RL agents?

In short, it is much easier for the agent to learn from a smaller dimensional state space. This is because the agent must also do representation learning; i.e. it must also infer what the state is ...
user avatar
3 votes
Accepted

How to determine the quality of synthetic data?

Due to subjective nature, quantitative evaluation of synthetic images is difficult in general. However, there are metrics like Inception Score or FID score that are used for evaluation of generative ...
user avatar
  • 238
2 votes

Can autoencoders be used for supervised learning?

One such paper I know of and which I implemented is Semi-Supervised Learning using Ladder Networks . I quote here their description of the model: Our approach follows Valpola (2015), who proposed a ...
user avatar
2 votes
Accepted

What are good parameters of an encoder?

I can not seem to grasp which specs make an encoder better than another one In general, in unsupervised settings, we want to learn the probability distribution of the data p(x) by some latent ...
user avatar
2 votes
Accepted

Do Le et al. (2012) train all three autoencoder layers at a time, or just one?

The paper refers to layers and sub-layers, and clearly indicates that one layer includes all three sub-layers, so when they say they train all three layers simultaneously, they are talking about the ...
user avatar
2 votes

Why isn't the Credit Card Fraud Detection dataset from Kaggle already balanced?

I believe that the idea is to have a similar ratio of fraud/"normal transaction" as to the ones that bank encounter on real life. If you balance it you will probably have a lot of false positive once ...
user avatar
  • 424
2 votes

What are the purposes of autoencoders?

PCA is a linear method that creates a transformation that is capable of changing the vectors projections (changing axis) Since PCA looks for the direction of maximum variance it usually have high ...
user avatar
2 votes
Accepted

How should I detect an object in a camera image?

Is this a good approach? Will I have a lot of trouble with different backgrounds? A lot will depend on the nature of the backgrounds you have, and how well they encode/decode by themselves without ...
user avatar
  • 23.9k
2 votes

Using ML to encypher data for production

It sounds like you are trying to compress data, and then recover the same data later. The most common tool for this task is an autoencoder. This model accepts data as input, and then learns to ...
user avatar
2 votes

How are small scale features represented in an Inverse Graphics Network (autoencoder)?

Simply said, there is no specific "meaning" to the features generated. They are simply features that are fitted through math and calculus, and nobody knows what they represent exactly, and will never ...
user avatar
  • 1,715
2 votes

Autoencoder produces repeated artifacts after convergence

Perhaps you are getting checkerboard artifacts Explained here, solutions involve changing the kernel and stride size to prevent them from being not divisible. Besides that, a solution could be to ...
user avatar
2 votes
Accepted

Why is exp used in encoder of VAE instead of using the value of standard deviation alone?

In the source code, the author defines sd by sd = 0.5 * tf.layers.dense(x, units=n_latent) which means that $\...
user avatar
  • 261
2 votes

In variational autoencoders, why do people use MSE for the loss?

On page 5 of the VAE paper, it's clearly stated We let $p_{\boldsymbol{\theta}}(\mathbf{x} \mid \mathbf{z})$ be a multivariate Gaussian (in case of real-valued data) or Bernoulli (in case of binary ...
user avatar
  • 33.8k
2 votes

Strange artifacts in autoencoder outputs

That's a classic checkerboard artifact. I would guess you're using a CNN as encoder/decoder architectures, since is well known that convolution layers, especially at the upsampling phase, cause these ...
user avatar
1 vote
Accepted

How to add some data input in a CNN?

If your interest is positional information, encode it! This could include learning an embedding for each position and leveraging that in your model. You could also use an approach to hard-encode ...
user avatar
  • 2,249
1 vote

What is the advantage of using a VAE over a deterministic auto-encoder?

It seems that you think that we want to perform classification with VAEs or that images that we pass to the encoder fall into more than one category. The other answer already points out that VAEs are ...
user avatar
  • 33.8k
1 vote

What can I do with an autoencoder?

An autoencoder learns to compress data, and then to decompress it again, recovering the original data. It does this by learning a mapping from the original feature space to a lower-dimensional space, ...
user avatar

Only top scored, non community-wiki answers of a minimum length are eligible