Questions tagged [autoencoders]

For questions about autoencoders, a type of unsupervised artificial network for learning efficient data codings.

Filter by
Sorted by
Tagged with
12 votes
4 answers
3k views

What are the purposes of autoencoders?

Autoencoders are neural networks that learn a compressed representation of the input in order to later reconstruct it, so they can be used for dimensionality reduction. They are composed of an encoder ...
user avatar
  • 34.2k
10 votes
1 answer
4k views

Loss jumps abruptly when I decay the learning rate with Adam optimizer in PyTorch

I'm training an auto-encoder network with Adam optimizer (with amsgrad=True) and ...
user avatar
10 votes
2 answers
3k views

Can autoencoders be used for supervised learning?

Can autoencoders be used for supervised learning without adding an output layer? Can we simply feed it with a concatenated input-output vector for training, and reconstruct the output part from the ...
user avatar
  • 2,029
8 votes
3 answers
2k views

What is the difference between encoders and auto-encoders?

How are the layers in a encoder connected across the network for normal encoders and auto-encoders? In general, what is the difference between encoders and auto-encoders?
user avatar
  • 303
8 votes
2 answers
8k views

Why is the variational auto-encoder's output blurred, while GANs output is crisp and has sharp edges?

I observed in several papers that the variational autoencoder's output is blurred, while GANs output is crisp and has sharp edges. Can someone please give some intuition why that is the case? I did ...
user avatar
  • 229
7 votes
1 answer
174 views

Are there transformer-based architectures that can produce fixed-length vector encodings given arbitrary-length text documents?

BERT encodes a piece of text such that each token (usually words) in the input text map to a vector in the encoding of the text. However, this makes the length of the encoding vary as a function of ...
user avatar
6 votes
2 answers
252 views

Why don't we use auto-encoders instead of GANs?

I have watched Stanford's lectures about artificial intelligence, I currently have one question: why don't we use autoencoders instead of GANs? Basically, what GAN does is it receives a random vector ...
user avatar
6 votes
4 answers
712 views

Is it possible for a neural network to be used to compress data?

When training a neural network, we often run into the issue of overfitting. However, is it possible to put overfitting to use? Basically, my idea is, instead of storing a large dataset in a database, ...
user avatar
  • 183
6 votes
2 answers
202 views

Is plain autoencoder a generative model?

I am wondering how a plain auto encoder is a generative model though its version might be but how can a plain auto encoder can be generative. I know that Vaes which is a version of the autoencoder is ...
user avatar
6 votes
1 answer
967 views

How should we choose the dimensions of the encoding layer in auto-encoders?

How should we choose the dimensions of the encoding layer in auto-encoders?
user avatar
  • 101
6 votes
0 answers
4k views

Does it make sense to use batch normalization in deep (stacked) or sparse auto-encoders?

Does it make sense to use batch normalization in deep (stacked) or sparse auto-encoders? I cannot find any resources for that. Is it safe to assume that, since it works for other DNNs, it will also ...
user avatar
  • 211
5 votes
1 answer
89 views

Concrete example of latent variables and observables plugged into the Bayes' rule

In the context of the variational auto-encoder, can someone give me a concrete example of the application of the Bayes' rule $$p_{\theta}(z|x)=\frac{p_{\theta}(x|z)p(z)}{p(x)}$$ for a given latent ...
user avatar
5 votes
1 answer
2k views

How to add a dense layer after a 2d convolutional layer in a convolutional autoencoder?

I am trying to implement a convolutional autoencoder with a dense layer at the bottleneck to do some dimensional reduction. I have seen two approaches for this, which aren't particularly scalable. The ...
user avatar
5 votes
1 answer
534 views

How can genetic programming be used in the context of auto-encoders?

I am trying to understand how genetic programming can be used in the context of auto-encoders. Currently, I am going through 2 papers Training Feedforward Neural Networks Using Genetic Algorithms (a ...
user avatar
  • 243
4 votes
1 answer
4k views

Why doesn't VAE suffer mode collapse?

Mode collapse is a common problem faced by GANs. I am curious why doesn't VAE suffer mode collapse?
user avatar
  • 229
4 votes
1 answer
338 views

Autoencoder produces repeated artifacts after convergence

As experiment, I have tried using an autoencoder to encode height data from the alps, however the decoded image is very pixellated after training for several hours as show in the image below. This ...
user avatar
  • 221
4 votes
1 answer
1k views

Variational Autoencoder task for better feature extraction

I have a CNN with the regression task of a single scalar. I was wondering if an additional task of reconstructing the image (used for learning visual concepts), seen in a DeepMind presentation with ...
user avatar
4 votes
2 answers
157 views

Can I apply reparametrization trick on "any" deep neural network?

I came across the "reparametrization trick" for the first time in the following paragraph from the chapter named Vector Calculus from the test book titled Mathematics for Machine Learning ...
user avatar
  • 3,191
4 votes
1 answer
128 views

What are some new deep learning models for learning latent representation of data?

I know that autoencoders are one type of deep neural networks that can learn the latent representation of data. I guess there should be several other models like autoencoders. What are some new deep ...
user avatar
  • 143
4 votes
1 answer
428 views

Is there a continuous conditional variational auto-encoder?

The Conditional Variational Autoencoder (CVAE), introduced in the paper Learning Structured Output Representation using Deep Conditional Generative Models (2015), is an extension of Variational ...
user avatar
  • 141
4 votes
0 answers
658 views

Is there any way and any reason why one would introduce a sparsity constraint on a deep auto-encoder?

Is there any way and any reason why one would introduce a sparsity constraint on a deep autoencoder? In particular, in deep autoencoders, the first layer often has more units than the dimensionality ...
user avatar
  • 577
3 votes
1 answer
139 views

How to determine the quality of synthetic data?

I'm working on a VAE model to produce synthetic data of X-Ray diffraction spectrums. I try to figure out how I can measure the quality of the spectrums. The goal would be to produce synthetic data ...
user avatar
  • 33
3 votes
1 answer
59 views

How are small scale features represented in an Inverse Graphics Network (autoencoder)?

This post refers to Fig. 1 of a paper by Microsoft on their Deep Convolutional Inverse Graphics Network: https://www.microsoft.com/en-us/research/wp-content/uploads/2016/11/kwkt_nips2015.pdf Having ...
user avatar
3 votes
1 answer
87 views

How should I detect an object in a camera image?

I would like to create a model, that will tell me if one type of object is in an image or not. So, for example, I have a camera and I would like to see when one object gets into the shot. Object ...
user avatar
3 votes
1 answer
217 views

How to evaluate the performance of an autoencoder trained on image data?

I am training an autoencoder on (general) image data. I use binary crossentropy loss function, but it is not very informative when I want to evaluate the performance of my autoencoder. An obvious ...
user avatar
  • 150
3 votes
2 answers
55 views

Dealing with empty frames in MRI images

I started working on the application of deep learning in medical imaging recently. While dealing with MRI images in the BraTS dataset, I observe that first and last few frames are always completely ...
user avatar
3 votes
1 answer
392 views

How can auto-encoders compute the reconstruction error for the new data?

Autoencoders are used for unsupervised anomaly detection by first learning the features of the data set with mainly "normal" data points. Then new data can be considered anomalous if the new ...
user avatar
  • 131
3 votes
0 answers
29 views

How to quantify the amount of information lost by the decoder NN in an AE?

Is there a way to quantify the amount of information lost in the lossy part of an autoencoder where the original input is compressed to a representation with less degrees of freedom? I was thinking ...
user avatar
  • 261
3 votes
0 answers
47 views

Enforcing sparsity constraints that make use of spatial contiguity

I have a deep learning network that outputs grayscale image reconstructions. In addition to good reconstruction performance (measured through mean squared error or some other measure like psnr), I ...
user avatar
3 votes
0 answers
56 views

Looking for the proper algorithm to compress many lowres images of nearby locations

I have an optimization problem that I'm looking for the right algorithm to solve. What I have: A large set of low-res 360 images that were taken on a regular grid within a certain area. each of these ...
user avatar
3 votes
0 answers
356 views

Are there any general tips for troubleshooting a VAE when apparently it is not learning?

I am trying to train a VAE for anomaly detection. I chose one architecture from this Github repository and I adjusted the input and output to match what I need. In my case, the input (and hence the ...
user avatar
  • 31
3 votes
2 answers
141 views

Do we also need to model a probability distribution for the decoder of a VAE?

I'm working on understanding VAEs, mostly through video lectures of Stanford cs231n, in particular lecture 13 tackles on this topic and I think I have a good theoretical grasp. However, when looking ...
user avatar
  • 325
2 votes
1 answer
91 views

How does replacing states with latent representations help RL agents?

I have seen many papers using autoencoders to replace images (states) with latent representations. Some of those methods have shown higher rewards using such techniques. However, I do not understand ...
user avatar
2 votes
1 answer
53 views

Using ML to encypher data for production

I am looking for research and experience working with ML models to ingest data for tasks, like text analysis, and creates a system that copies (or in other words enciphers) the input data, to then ...
user avatar
  • 348
2 votes
1 answer
142 views

Why isn't the Credit Card Fraud Detection dataset from Kaggle already balanced?

I am working on a credit card fraud detection problem using autoencoders. I have a question regarding the dataset I'll be using. I've downloaded the dataset for the above problem from Kaggle, which ...
user avatar
  • 101
2 votes
1 answer
64 views

Do Le et al. (2012) train all three autoencoder layers at a time, or just one?

Le et al. 2012 use a network of 1 billion parameters to learn neurons that respond to faces, cats, pedestrians, etc. without labels (unsupervised). Their network is built with three autoregressive ...
user avatar
2 votes
2 answers
690 views

How can I have the same input and output shape in an auto-encoder?

I'm building a denoising autoencoder. I want to have the same input and output shape image. This is my architecture: ...
user avatar
2 votes
1 answer
59 views

What is the mean in the variational auto-encoder?

Here's a diagram of a variational auto-encoder. There are 2 nodes before the sample (encoding vector). One is the mean, one is the standard deviation. The mean one is confusing. Is it the mean of ...
user avatar
  • 1,203
2 votes
1 answer
220 views

Autoencoder network for feature selection not converging

I am training an undercomplete autoencoder network for feature selection. I am using one hidden layer in the encoder and decoder networks each. The ELU activation function is used for each layer. For ...
user avatar
2 votes
1 answer
198 views

How to add some data input in a CNN?

There is this problem I have encountered, I was trying to classify the pixels from input image into classes, sort of like segmentation, using a encoder-decoder CNN. The “interested” pixels usually ...
user avatar
  • 23
2 votes
1 answer
208 views

VAE giving near zero output when latent space dimension is large

I'm training a VAE to reconstruct some input (channels picked up by some MIMO BS for context) and I ran an experiment on the training set to see how the performance improves with the latent space ...
user avatar
2 votes
0 answers
32 views

Literature on the advantages of using an auto-encoder for classification

Given a supervised problem with X, y input pairs, one can do two things for obtaining the function f that maps X with y with Neural Networks (and in general in machine learning): Deploy directly a ...
user avatar
2 votes
0 answers
31 views

Compressing Parameters of an Response System

I have an input-output system, which is fully determined by 256 parameters, of which I know a significant amount are of less importance to the input-output pattern. The data I have is some (64k in ...
user avatar
  • 121
2 votes
1 answer
123 views

Are Autoencoders for noise-reduction only suited to deal with salt-and-pepper kind of noise?

I'm currently looking at NN to deal with noisy data. I like the Autoencoder approach https://medium.com/@aliaksei.mikhailiuk/unsupervised-learning-for-data-interpolation-e259cf5dc957 because it seems ...
user avatar
2 votes
1 answer
220 views

What are the main differences between sparse autoencoders and convolution autoencoders?

What are the main differences and similarities between sparse autoencoders and convolution autoencoders? When should one be preferred over the other? What are their applications? (References are ...
user avatar
  • 1,193
2 votes
2 answers
214 views

How can I use autoencoders to analyze patterns and classify them?

I generated a bunch of simulation data from a complex physical simulation that spits out patterns. I am trying to apply unsupervised learning to analyze the patterns and ideally classify them into ...
user avatar
2 votes
2 answers
687 views

Why does the denoising autoencoder always returns the same output?

I am trying to implement a denoising autoencoder (DAE) to remove noise from 1024-point FFT spectra. I am using two types of spectra: (1) that contain a distinctive high amplitude spectral peak and (2) ...
user avatar
2 votes
0 answers
613 views

How does deepfake technology work with multiple people in a single frame?

I was watching this video from corridor crew, according to them, they have used deepfake technology to create this video. I myself have never made a deepfake videos, but I have enough knowledge in the ...
user avatar
  • 946
2 votes
1 answer
121 views

What are some limitations of using Collaborative Deep learning for Recommender systems?

Recently I worked on a paper by Hao Wang, Collaborative Deep learning for Recommender Systems; which uses a two way tightly coupled method, Collaborative filtering for Item correlation and Stacked ...
user avatar
  • 303
1 vote
1 answer
205 views

Why is exp used in encoder of VAE instead of using the value of standard deviation alone?

There's one VAE example here: https://towardsdatascience.com/teaching-a-variational-autoencoder-vae-to-draw-mnist-characters-978675c95776. And the source code of encoder can be found at the ...
user avatar
  • 1,203