Questions tagged [autoencoders]

For questions about autoencoders, a type of unsupervised artificial network for learning efficient data codings.

Filter by
Sorted by
Tagged with
12 votes
4 answers
3k views

What are the purposes of autoencoders?

Autoencoders are neural networks that learn a compressed representation of the input in order to later reconstruct it, so they can be used for dimensionality reduction. They are composed of an encoder ...
nbro's user avatar
  • 40.6k
10 votes
1 answer
4k views

Loss jumps abruptly when I decay the learning rate with Adam optimizer in PyTorch

I'm training an auto-encoder network with Adam optimizer (with amsgrad=True) and ...
imflash217's user avatar
10 votes
3 answers
2k views

What is the difference between encoders and auto-encoders?

How are the layers in a encoder connected across the network for normal encoders and auto-encoders? In general, what is the difference between encoders and auto-encoders?
m2rik's user avatar
  • 333
10 votes
2 answers
3k views

Can autoencoders be used for supervised learning?

Can autoencoders be used for supervised learning without adding an output layer? Can we simply feed it with a concatenated input-output vector for training, and reconstruct the output part from the ...
rcpinto's user avatar
  • 2,119
9 votes
2 answers
2k views

Is plain autoencoder a generative model?

I am wondering how a plain auto encoder is a generative model though its version might be but how can a plain auto encoder can be generative. I know that Vaes which is a version of the autoencoder is ...
Nervous Hero's user avatar
9 votes
4 answers
12k views

Why is the variational auto-encoder's output blurred, while GANs output is crisp and has sharp edges?

I observed in several papers that the variational autoencoder's output is blurred, while GANs output is crisp and has sharp edges. Can someone please give some intuition why that is the case? I did ...
Trect's user avatar
  • 269
8 votes
2 answers
858 views

Are there transformer-based architectures that can produce fixed-length vector encodings given arbitrary-length text documents?

BERT encodes a piece of text such that each token (usually words) in the input text map to a vector in the encoding of the text. However, this makes the length of the encoding vary as a function of ...
HelloGoodbye's user avatar
8 votes
1 answer
4k views

Does it make sense to use batch normalization in deep (stacked) or sparse auto-encoders?

Does it make sense to use batch normalization in deep (stacked) or sparse auto-encoders? I cannot find any resources for that. Is it safe to assume that, since it works for other DNNs, it will also ...
Glrs's user avatar
  • 231
7 votes
1 answer
6k views

Why doesn't VAE suffer mode collapse?

Mode collapse is a common problem faced by GANs. I am curious why doesn't VAE suffer mode collapse?
Trect's user avatar
  • 269
6 votes
2 answers
438 views

Why don't we use auto-encoders instead of GANs?

I have watched Stanford's lectures about artificial intelligence, I currently have one question: why don't we use autoencoders instead of GANs? Basically, what GAN does is it receives a random vector ...
dato nefaridze's user avatar
6 votes
4 answers
1k views

Is it possible for a neural network to be used to compress data?

When training a neural network, we often run into the issue of overfitting. However, is it possible to put overfitting to use? Basically, my idea is, instead of storing a large dataset in a database, ...
Bryan Tan's user avatar
  • 183
6 votes
1 answer
2k views

How should we choose the dimensions of the encoding layer in auto-encoders?

How should we choose the dimensions of the encoding layer in auto-encoders?
Neha soni's user avatar
  • 101
5 votes
1 answer
103 views

Concrete example of latent variables and observables plugged into the Bayes' rule

In the context of the variational auto-encoder, can someone give me a concrete example of the application of the Bayes' rule $$p_{\theta}(z|x)=\frac{p_{\theta}(x|z)p(z)}{p(x)}$$ for a given latent ...
user8714896's user avatar
5 votes
1 answer
608 views

Autoencoder produces repeated artifacts after convergence

As experiment, I have tried using an autoencoder to encode height data from the alps, however the decoded image is very pixellated after training for several hours as show in the image below. This ...
Yadeses's user avatar
  • 231
5 votes
1 answer
3k views

How to add a dense layer after a 2d convolutional layer in a convolutional autoencoder?

I am trying to implement a convolutional autoencoder with a dense layer at the bottleneck to do some dimensional reduction. I have seen two approaches for this, which aren't particularly scalable. The ...
Pavan Inguva's user avatar
5 votes
1 answer
639 views

How can genetic programming be used in the context of auto-encoders?

I am trying to understand how genetic programming can be used in the context of auto-encoders. Currently, I am going through 2 papers Training Feedforward Neural Networks Using Genetic Algorithms (a ...
Gabriele's user avatar
  • 243
4 votes
2 answers
1k views

Can I apply reparametrization trick on "any" deep neural network?

I came across the "reparametrization trick" for the first time in the following paragraph from the chapter named Vector Calculus from the test book titled Mathematics for Machine Learning ...
hanugm's user avatar
  • 3,840
4 votes
1 answer
2k views

Variational Autoencoder task for better feature extraction

I have a CNN with the regression task of a single scalar. I was wondering if an additional task of reconstructing the image (used for learning visual concepts), seen in a DeepMind presentation with ...
Florentin Alexandru Iftimie's user avatar
4 votes
1 answer
1k views

How to evaluate the performance of an autoencoder trained on image data?

I am training an autoencoder on (general) image data. I use binary crossentropy loss function, but it is not very informative when I want to evaluate the performance of my autoencoder. An obvious ...
nim.py's user avatar
  • 160
4 votes
1 answer
153 views

What are some new deep learning models for learning latent representation of data?

I know that autoencoders are one type of deep neural networks that can learn the latent representation of data. I guess there should be several other models like autoencoders. What are some new deep ...
Kadaj13's user avatar
  • 143
4 votes
1 answer
760 views

Is there a continuous conditional variational auto-encoder?

The Conditional Variational Autoencoder (CVAE), introduced in the paper Learning Structured Output Representation using Deep Conditional Generative Models (2015), is an extension of Variational ...
D1X's user avatar
  • 141
4 votes
0 answers
675 views

Is there any way and any reason why one would introduce a sparsity constraint on a deep auto-encoder?

Is there any way and any reason why one would introduce a sparsity constraint on a deep autoencoder? In particular, in deep autoencoders, the first layer often has more units than the dimensionality ...
MattSt's user avatar
  • 597
3 votes
2 answers
7k views

In variational autoencoders, why do people use MSE for the loss?

In VAEs, we try to maximize the ELBO = $\mathbb{E}_q [\log\ p(x|z)] + D_{KL}(q(z \mid x), p(z))$, but I see that many implement the first term as the MSE of the image and its reconstruction. Here's a ...
IttayD's user avatar
  • 219
3 votes
1 answer
349 views

How to determine the quality of synthetic data?

I'm working on a VAE model to produce synthetic data of X-Ray diffraction spectrums. I try to figure out how I can measure the quality of the spectrums. The goal would be to produce synthetic data ...
Tekay's user avatar
  • 33
3 votes
1 answer
73 views

How are small scale features represented in an Inverse Graphics Network (autoencoder)?

This post refers to Fig. 1 of a paper by Microsoft on their Deep Convolutional Inverse Graphics Network: https://www.microsoft.com/en-us/research/wp-content/uploads/2016/11/kwkt_nips2015.pdf Having ...
natanijelvasic's user avatar
3 votes
1 answer
132 views

How should I detect an object in a camera image?

I would like to create a model, that will tell me if one type of object is in an image or not. So, for example, I have a camera and I would like to see when one object gets into the shot. Object ...
Marko Zadravec's user avatar
3 votes
1 answer
219 views

Are Autoencoders for noise-reduction only suited to deal with salt-and-pepper kind of noise?

I'm currently looking at NN to deal with noisy data. I like the Autoencoder approach https://medium.com/@aliaksei.mikhailiuk/unsupervised-learning-for-data-interpolation-e259cf5dc957 because it seems ...
matthias_buehlmann's user avatar
3 votes
2 answers
91 views

Dealing with empty frames in MRI images

I started working on the application of deep learning in medical imaging recently. While dealing with MRI images in the BraTS dataset, I observe that first and last few frames are always completely ...
Suyog Jadhav's user avatar
3 votes
1 answer
809 views

How can auto-encoders compute the reconstruction error for the new data?

Autoencoders are used for unsupervised anomaly detection by first learning the features of the data set with mainly "normal" data points. Then new data can be considered anomalous if the new ...
Brian's user avatar
  • 131
3 votes
0 answers
1k views

Why VQ-VAE instead of VAE?

From the paper on VQ-VAE, it said that the vector quantized variational autoencoder (VQ-VAE), differs from VAEs in two key ways: the encoder network output discrete, rather than continuous, codecs ...
Nervous Hero's user avatar
3 votes
0 answers
54 views

How to quantify the amount of information lost by the decoder NN in an AE?

Is there a way to quantify the amount of information lost in the lossy part of an autoencoder where the original input is compressed to a representation with less degrees of freedom? I was thinking ...
0x90's user avatar
  • 281
3 votes
0 answers
64 views

Enforcing sparsity constraints that make use of spatial contiguity

I have a deep learning network that outputs grayscale image reconstructions. In addition to good reconstruction performance (measured through mean squared error or some other measure like psnr), I ...
Jane Sully's user avatar
3 votes
0 answers
57 views

Looking for the proper algorithm to compress many lowres images of nearby locations

I have an optimization problem that I'm looking for the right algorithm to solve. What I have: A large set of low-res 360 images that were taken on a regular grid within a certain area. each of these ...
matthias_buehlmann's user avatar
3 votes
0 answers
1k views

How does deepfake technology work with multiple people in a single frame?

I was watching this video from corridor crew, according to them, they have used deepfake technology to create this video. I myself have never made a deepfake videos, but I have enough knowledge in the ...
Eka's user avatar
  • 1,066
3 votes
0 answers
476 views

Are there any general tips for troubleshooting a VAE when apparently it is not learning?

I am trying to train a VAE for anomaly detection. I chose one architecture from this Github repository and I adjusted the input and output to match what I need. In my case, the input (and hence the ...
Bill's user avatar
  • 31
3 votes
2 answers
220 views

Do we also need to model a probability distribution for the decoder of a VAE?

I'm working on understanding VAEs, mostly through video lectures of Stanford cs231n, in particular lecture 13 tackles on this topic and I think I have a good theoretical grasp. However, when looking ...
ytolochko's user avatar
  • 365
2 votes
2 answers
5k views

What is an appropriate size for a latent space of (variational) autoencoders and how it varies with the features of the images?

I am training an autoencoder and a variational autoencoder using satellite and streetview images. I have tested my program on standard datasets such as MNIST and CelebA. It seems that the latent space ...
Qingyi Wang's user avatar
2 votes
1 answer
138 views

How does replacing states with latent representations help RL agents?

I have seen many papers using autoencoders to replace images (states) with latent representations. Some of those methods have shown higher rewards using such techniques. However, I do not understand ...
desert_ranger's user avatar
2 votes
1 answer
103 views

Why can Variational Autoencoders (VAEs) approximate arbitrary distributions?

I am trying to reason to myself why is it that VAEs can approximate arbitrary probability distributions even though 𝑞𝜙(𝑧|𝑥) and 𝑝𝜃(𝑥|𝑧) are Gaussian. I understand that the parameters are ...
Joel's user avatar
  • 33
2 votes
1 answer
55 views

Using ML to encypher data for production

I am looking for research and experience working with ML models to ingest data for tasks, like text analysis, and creates a system that copies (or in other words enciphers) the input data, to then ...
benbyford's user avatar
  • 348
2 votes
1 answer
156 views

Why isn't the Credit Card Fraud Detection dataset from Kaggle already balanced?

I am working on a credit card fraud detection problem using autoencoders. I have a question regarding the dataset I'll be using. I've downloaded the dataset for the above problem from Kaggle, which ...
Neha soni's user avatar
  • 101
2 votes
1 answer
75 views

Do Le et al. (2012) train all three autoencoder layers at a time, or just one?

Le et al. 2012 use a network of 1 billion parameters to learn neurons that respond to faces, cats, pedestrians, etc. without labels (unsupervised). Their network is built with three autoregressive ...
Josiah Yoder's user avatar
2 votes
2 answers
1k views

How can I have the same input and output shape in an auto-encoder?

I'm building a denoising autoencoder. I want to have the same input and output shape image. This is my architecture: ...
Vesko Vujovic's user avatar
2 votes
1 answer
179 views

What is the mean in the variational auto-encoder?

Here's a diagram of a variational auto-encoder. There are 2 nodes before the sample (encoding vector). One is the mean, one is the standard deviation. The mean one is confusing. Is it the mean of ...
Dan D.'s user avatar
  • 1,283
2 votes
1 answer
333 views

Autoencoder network for feature selection not converging

I am training an undercomplete autoencoder network for feature selection. I am using one hidden layer in the encoder and decoder networks each. The ELU activation function is used for each layer. For ...
Prishita Ray's user avatar
2 votes
1 answer
394 views

How to add some data input in a CNN?

There is this problem I have encountered, I was trying to classify the pixels from input image into classes, sort of like segmentation, using a encoder-decoder CNN. The “interested” pixels usually ...
kdlsw's user avatar
  • 23
2 votes
1 answer
808 views

VAE giving near zero output when latent space dimension is large

I'm training a VAE to reconstruct some input (channels picked up by some MIMO BS for context) and I ran an experiment on the training set to see how the performance improves with the latent space ...
tyassine's user avatar
2 votes
0 answers
36 views

Literature on the advantages of using an auto-encoder for classification

Given a supervised problem with X, y input pairs, one can do two things for obtaining the function f that maps X with y with Neural Networks (and in general in machine learning): Deploy directly a ...
Tommaso Bendinelli's user avatar
2 votes
0 answers
32 views

Compressing Parameters of an Response System

I have an input-output system, which is fully determined by 256 parameters, of which I know a significant amount are of less importance to the input-output pattern. The data I have is some (64k in ...
t-smart's user avatar
  • 121
2 votes
1 answer
555 views

What are the main differences between sparse autoencoders and convolution autoencoders?

What are the main differences and similarities between sparse autoencoders and convolution autoencoders? When should one be preferred over the other? What are their applications? (References are ...
Pluviophile's user avatar
  • 1,273