Questions tagged [autoencoders]

For questions about autoencoders, a type of unsupervised artificial network for learning efficient data codings.

51 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
7 votes
1 answer
175 views

Are there transformer-based architectures that can produce fixed-length vector encodings given arbitrary-length text documents?

BERT encodes a piece of text such that each token (usually words) in the input text map to a vector in the encoding of the text. However, this makes the length of the encoding vary as a function of ...
user avatar
6 votes
0 answers
4k views

Does it make sense to use batch normalization in deep (stacked) or sparse auto-encoders?

Does it make sense to use batch normalization in deep (stacked) or sparse auto-encoders? I cannot find any resources for that. Is it safe to assume that, since it works for other DNNs, it will also ...
user avatar
  • 211
4 votes
0 answers
658 views

Is there any way and any reason why one would introduce a sparsity constraint on a deep auto-encoder?

Is there any way and any reason why one would introduce a sparsity constraint on a deep autoencoder? In particular, in deep autoencoders, the first layer often has more units than the dimensionality ...
user avatar
  • 577
3 votes
0 answers
29 views

How to quantify the amount of information lost by the decoder NN in an AE?

Is there a way to quantify the amount of information lost in the lossy part of an autoencoder where the original input is compressed to a representation with less degrees of freedom? I was thinking ...
user avatar
  • 261
3 votes
0 answers
47 views

Enforcing sparsity constraints that make use of spatial contiguity

I have a deep learning network that outputs grayscale image reconstructions. In addition to good reconstruction performance (measured through mean squared error or some other measure like psnr), I ...
user avatar
3 votes
0 answers
56 views

Looking for the proper algorithm to compress many lowres images of nearby locations

I have an optimization problem that I'm looking for the right algorithm to solve. What I have: A large set of low-res 360 images that were taken on a regular grid within a certain area. each of these ...
user avatar
3 votes
0 answers
358 views

Are there any general tips for troubleshooting a VAE when apparently it is not learning?

I am trying to train a VAE for anomaly detection. I chose one architecture from this Github repository and I adjusted the input and output to match what I need. In my case, the input (and hence the ...
user avatar
  • 31
2 votes
1 answer
209 views

VAE giving near zero output when latent space dimension is large

I'm training a VAE to reconstruct some input (channels picked up by some MIMO BS for context) and I ran an experiment on the training set to see how the performance improves with the latent space ...
user avatar
2 votes
0 answers
32 views

Literature on the advantages of using an auto-encoder for classification

Given a supervised problem with X, y input pairs, one can do two things for obtaining the function f that maps X with y with Neural Networks (and in general in machine learning): Deploy directly a ...
user avatar
2 votes
0 answers
31 views

Compressing Parameters of an Response System

I have an input-output system, which is fully determined by 256 parameters, of which I know a significant amount are of less importance to the input-output pattern. The data I have is some (64k in ...
user avatar
  • 121
2 votes
1 answer
123 views

Are Autoencoders for noise-reduction only suited to deal with salt-and-pepper kind of noise?

I'm currently looking at NN to deal with noisy data. I like the Autoencoder approach https://medium.com/@aliaksei.mikhailiuk/unsupervised-learning-for-data-interpolation-e259cf5dc957 because it seems ...
user avatar
2 votes
2 answers
215 views

How can I use autoencoders to analyze patterns and classify them?

I generated a bunch of simulation data from a complex physical simulation that spits out patterns. I am trying to apply unsupervised learning to analyze the patterns and ideally classify them into ...
user avatar
2 votes
0 answers
613 views

How does deepfake technology work with multiple people in a single frame?

I was watching this video from corridor crew, according to them, they have used deepfake technology to create this video. I myself have never made a deepfake videos, but I have enough knowledge in the ...
user avatar
  • 946
2 votes
1 answer
121 views

What are some limitations of using Collaborative Deep learning for Recommender systems?

Recently I worked on a paper by Hao Wang, Collaborative Deep learning for Recommender Systems; which uses a two way tightly coupled method, Collaborative filtering for Item correlation and Stacked ...
user avatar
  • 303
1 vote
0 answers
40 views

Is there a name for this model?

I have an image autoencoder model trained as follows: Step 1) train a GAN to obtain a generator capable of drawing from the data manifold by sampling a normal distribution in latent space Step 2) ...
user avatar
1 vote
1 answer
35 views

Why is training all layers at a time effective for a multi-layer autoencoder?

This training of all layers of a CNN simultaneously is standard practice today. It is found in every CNN (AlexNet (2012), VGG, Inception, GANs, etc) and even pre-CNN networks such as Le et al. 2012. ...
user avatar
1 vote
0 answers
35 views

What is the conceptual difference between convolutional neural networks and auto-encoders?

I'm familiar with Auto-Encoders and I'm about to dive into CNNs. By having a look at the most important component of a CNN, the filter: I wonder how it is different from Auto-Encoders: For me, it ...
user avatar
  • 205
1 vote
0 answers
35 views

How do I select the number of neurons for each layer in an auto-encoder for dimensionality reduction?

I am trying to apply an auto-encoder for dimensionality reduction. I wonder how it will be applied on a large dataset. I have tried this code below. I have total of 8 features in my data and I want to ...
user avatar
  • 111
1 vote
0 answers
20 views

Dealing with bias in multi-channel auto encoders

The problem I have a multi-channel 1D signal I want to auto-encode. I am unable to resonstruct the input when the number of channels increases. Code I am using a convolutional encoder, and a ...
user avatar
  • 669
1 vote
0 answers
53 views

Underfitting a single batch: Can't cause autoencoder to overfit multi-sample batches of 1d data. How to debug?

TL;DR I am unable to overfit batches with multiple samples using autoencoder. Fully connected decoder seems to handle more samples per batch than conv decoder, but then also fails when number of ...
user avatar
  • 669
1 vote
0 answers
57 views

variational auto encoder loss goes down but does not reconstruct input. out of debugging ideas

My variational autoencoder seems to work for MNIST, but fails on slightly "harder" data. By "fails" I mean there are at least two apparent problems: Very poor reconstruction, for ...
user avatar
  • 669
1 vote
0 answers
35 views

How can I improve the performance on unseen data for semantic segmentation using an auto-encoder?

I am using simple autoencoders for the task of semantic segmentation on the VOC2012 dataset. I am currently using a simple autoencoder based model. It is trained on adam optimizer with cross-entropy ...
user avatar
1 vote
0 answers
51 views

how to handle highly imbalanced multilabel classification?

I am working on a multilabel classification in which I am having 206 labels. When I saw the percentage of the number of 1's in each label they are way less than 0.1% for each label. The maximum ...
user avatar
1 vote
0 answers
73 views

Role of autoencoder in Hierarchical Extreme Learning Machine

I want to build HELM neural network that consists of autoencoder (AE) and one class classification (OC). HELM with AE and OC have following shape: That is, hidden layer output of AE is input of OC. ...
user avatar
  • 11
1 vote
0 answers
81 views

How can a de-noising auto-encoder act as an anomaly detection model?

In some research papers, I have seen that, for training the autoencoders, instead of giving the non-anomalous input images, they add some anomalies to the normal input images, and train the auto-...
user avatar
  • 11
1 vote
0 answers
39 views

How much data do we need for making a successful de-noising auto-encoder?

Is there a guide how much data do you need for making successful denoising model using autoencoders? Or the rule is, the more data, the better it is? I tried with small dataset 350 samples, to see ...
user avatar
1 vote
0 answers
58 views

Does the reduction of the dimensions over multiple layers allow more details to be stored within the final representation?

From : https://debuggercafe.com/implementing-deep-autoencoder-in-pytorch/ the following autoencoder is defined ...
user avatar
  • 295
1 vote
0 answers
43 views

How estimate the minimum size of an autoencoder to overfit the training data?

Given e.g. $1$M vectors of $1000$ floating points each, where every point in vectors is sampled from a uniform distribution between $-1$ to $1$, how to estimate the minimum network size required ...
user avatar
  • 21
1 vote
0 answers
26 views

Can denoising auto-encoders be convolutional and fully connected?

I have been reading lately on autoencoders a lot. I just wanted to summarize my understanding of denoising autoencoders. As far as I understand they can be Fully connected (in which case, they will ...
user avatar
  • 111
1 vote
0 answers
103 views

Why is my variational auto-encoder generating random noise?

This is my first variational autoencoder. Background info: I am using the MNIST digits dataset. The model is created and trained in PyTorch. The model is able to get a reasonably low loss, but the ...
user avatar
  • 11
1 vote
0 answers
31 views

Reduce same sample distance in VAE encodings

I'm working on a beta VAE model learning a latent representation used as a similarity metric for image registration. One of the main problems I'm facing is that the encoder + sampler output doesn't ...
user avatar
  • 11
1 vote
0 answers
17 views

Camera pose to environment Mapping

I would like to teach a model the environment of a room. I'm doing so by mapping a camera pose (x, y, z, q0, q1, q2, q3) to its corresponding image; where x, y, z represent location in Cartesian ...
user avatar
  • 11
1 vote
0 answers
19 views

Limits for a bottleneck

I have some 64x64 pixels frames from a (simulated) video, with a spaceship moving on a fixed background. The spaceship moves in a straight line with constant velocity from left to right (along the x-...
user avatar
1 vote
0 answers
41 views

How to learn to sample?

Imagine you have access to a dataset of pairs $(s, \hat{\pi}(s))$ where s is a state in a high dimension continuous space $S$, $\pi(s)$ is a probabilistic distribution on a large discrete space $D$ (...
user avatar
  • 111
1 vote
0 answers
95 views

Can we use Autoencoders for unsupervised CNN feature learning?

I searched through the internet but couldn't find a reliable article that answers this question. Can we use Autoencoders for unsupervised CNN feature learning of unlabeled images like the below and ...
user avatar
0 votes
0 answers
9 views

Extracting behavior (switch On/Off) of an electric load from unlabeled time series data

Following are the details of my dataset: sampling frequency: 1 Hz No. of useful features: 10 The time series dataset is from household wherein I'm required to find ...
user avatar
0 votes
1 answer
45 views

Reconstructing 3D models from 2D images using autoencoders

I went through a research paper ("Voxel-Based 3D Object Reconstruction from Single 2D Image Using Variational Autoencoders") and tried to implement the approach following this diagram: ![...
user avatar
0 votes
1 answer
31 views

Autoencoders: Where does the encoder end and the decoder begin?

Consider a simple Autoencoder neural net: ...
user avatar
0 votes
0 answers
30 views

Why does the feature space of an autoencoder typically contain more info than a teacher-student model?

This is a question our Prof gave us as exam preparation, but I don't know why the Autoencoder should contain more info than the Teacher Student model. Teacher Student Models are a class of models in ...
user avatar
  • 348
0 votes
0 answers
19 views

Do channels on the encoder's output layer really matter in convolutional autoencoders?

I viewed the example in Keras for a convolutional autoencoder. I noticed that the number of units in each layer of both the encoder and decoder never falls below that of the input. In other words, ...
user avatar
0 votes
0 answers
59 views

VAEs vs Autoencoders with BatchNorm and Dropout?

It struck me that regular auto-encoders with batch-norm and dropout have quite similar properties to VAEs which made me wonder whether VAEs where really much better than this simpler alternative. Let ...
user avatar
  • 296
0 votes
1 answer
15 views

Do Quo et al (2013) perform backpropagation between layers?

Le et al. 2013's non-weight sharing CNN has inspired me to ask two questions on this site previously. When training the three-layer autoencoder, do they compute dL/dW (where L is equation 1) ...
user avatar
0 votes
1 answer
205 views

What is the state of the art concerning autoencoder that connect 2 images that are not similar but are physicaly related?

I am currently working on an autoencoder that connect two images. The first one can be seen as the electron flow and the second one is the electrostatic potential seen by the electrons. Long story ...
user avatar
0 votes
0 answers
30 views

Masked Autoencoder Structure

In the following structure when we use MADE due to the constraints for making a masked autoencoder, it seems some inputs do not have any connection to the next layer, and also there is the output that ...
user avatar
  • 1
0 votes
0 answers
29 views

Adversarial Autoencoder is not working and not learning properly

I am trying to get an Adversarial AutoEncoder going using keras Fit method on a keras.model class but for some reason it is not working. Keep in mind that I tried updating encoder and decoder at the ...
user avatar
  • 1
0 votes
0 answers
31 views

Train separate AutoEncoder's on each class or one AE for all classes to learn features?

I'm working on a project where the dataset contains time series of three classes, depending on the shape of the series. I want to learn the representations of these series as vectors, so naturally I ...
user avatar
0 votes
0 answers
34 views

Weird KL divergence behaviour

I'm training a complex model for motion prediction using a VAE, however the KL divergence has a very strange behavior. A scheleton of the network is the following: At the end my network compute the ...
user avatar
  • 159
0 votes
0 answers
130 views

Why would an auto-encoder produce latent vectors with many zeros?

My autoencoder give latent vectors with many zeroes components like: ...
user avatar
0 votes
0 answers
130 views

What is the difference between the forward pass of the Multi-Layer Perceptron, Deep AutoEncoder and Deep Belief Network?

Multi-Layer Perceptron (MLP), Deep AutoEncoder (DAE), and Deep Belief Network (DBN) are trained differently. However, do they follow the same process during the inference phase, i.e., do they ...
user avatar
  • 73
0 votes
0 answers
186 views

What is the time complexity for training a single-hidden layer auto-encoder?

What is the time complexity for training a single-hidden layer auto-encoder, for 1 epoch? You can assume that there are $n$ training examples, $m$ features, and $k$ neurons in the hidden layer, and ...
user avatar