10

I see no reason why decaying learning rates should create the kinds of jumps in losses that you are observing. It should "slow down" how quickly you "move", which in the case of a loss that otherwise consistently shrinks really should, at worst, just lead to a plateau in your losses (rather than those jumps). The first thing I observe in your code is that ...


6

It is important to think about what sort of patterns in the data are being represented. Suppose that you have a dataset of greyscale images, such that every image is a uniform intensity. As a human brain you'd realise that every element in this dataset can be described in terms of a single numeric parameter, which is that intensity value. This is something ...


5

Theory Encoder In general, an Encoder is a mapping $f : X \rightarrow Y $ with $X$ Input Space and $Y$ Code Space In case of Neural Networks, it is a Generative Model hence a function which is able to compute a Representation out of some input (like GAN) The point is: how would you train such an encoder network ? The general answer is: it depends on ...


4

The number of dimensions is a hyperparameter of your model, and you should do a hyperparameter search, like with any other parameters. There's also a tradeoff between dimension and training speed, so it should be small enough to be trainable in a reasonable time.


4

The key is: VAE usually use a small latent dimension, the information of input is so hard to pass through this bottleneck, meanwhile it tries to minimize the loss with the batch of input data, you should know the result -- VAE can only have a mean and blurry output. If you increase the bandwidth of the bottleneck, i.e. the size of latent vector, VAE can get ...


4

Let's assume the probability distributions are Gaussian (or normal) distributions. In other words, in the Bayes' rule \begin{align} p(z|x)=\frac{p(x|z)p(z)}{p(x)} \tag{1}\label{1} \end{align} The posterior $p(z|x)$, the likelihood $p(x|z)$, the prior $p(z)$ and the evidence (or marginal) $p(x)$ are Gaussian distributions. You can assume this because ...


4

In fact, autoencoders are used for generative tasks. Have a look at Tutorial on Variational Autoencoders (VAEs). The coolest thing about VAE is that abstract features can be easily amplified or suppressed based on extracted vectors from the latent space. Let's imagine a model trained on MNIST to generate digits. If you take two images of the same digit which ...


4

Auto-encoders are widely used and maybe even more used than GANs (in fact, auto-encoders are older than GANs, although the main general idea behind GANs is quite old). For example, auto-encoders are used in World Models, for drug design (e.g. see this paper) and many other tasks that involve data compression or generation. So, if we train autoencoders, for ...


3

A use case of autoencoders (in particular, of the decoder or generative model of the autoencoder) is to denoise the input. This type of autoencoders, called denoising autoencoders, take a partially corrupted input and they attempt to reconstruct the corresponding uncorrupted input. There are several applications of this model. For example, if you had a ...


3

In essence, Variational Autoencoders learn an "explicit" distribution of the data by trying to fit the data via a multi-dimensional Gaussian/Normal distribution. However, Generative Adversarial Networks learn an "implicit" distribution of data meaning that you cannot directly sample them. Also, due to the deterministic nature of neural networks GANs tend to ...


3

With Generative Adversarial Networks, all the generator cares about is fooling the discriminator. There's no requirement to be clever, or exhaustive, or make efficient use of the input space. As long as the discriminator returns "real" (vs. "fake") the generator "wins". The hope is that as the generator and discriminator are trained simultaneously, each ...


3

In short, it is much easier for the agent to learn from a smaller dimensional state space. This is because the agent must also do representation learning; i.e. it must also infer what the state is telling it as part of the learning process. If you think of the architecture used in DQN to solve Atari, they had a CNN that outputted a vector which was then ...


2

I can not seem to grasp which specs make an encoder better than another one In general, in unsupervised settings, we want to learn the probability distribution of the data p(x) by some latent variables that explain the variations observed in the training set. The autoencoder family (Variational, Denoising, Contrastive, Sparse) try to approximate p(x) ...


2

The paper refers to layers and sub-layers, and clearly indicates that one layer includes all three sub-layers, so when they say they train all three layers simultaneously, they are talking about the three autoencoder layers, not the sub-layers. This also agrees with the fact that only the autoencoder layer has tunable parameters. The other two layers use ...


2

I believe that the idea is to have a similar ratio of fraud/"normal transaction" as to the ones that bank encounter on real life. If you balance it you will probably have a lot of false positive once you apply your solution to real world's data and, if that may be fine for you to play with, it's not what a bank would like as they can't block too much of the ...


2

PCA is a linear method that creates a transformation that is capable of changing the vectors projections (changing axis) Since PCA looks for the direction of maximum variance it usually have high discriminativity BUT it does not guaranteed that the direction of most variance is the direction of most discriminativity. LDA is a linear method that creates a ...


2

Is this a good approach? Will I have a lot of trouble with different backgrounds? A lot will depend on the nature of the backgrounds you have, and how well they encode/decode by themselves without the object in frame. My gut feeling is that your system will have poor performance compared to a properly trained classifier, as the autoencoder will naturally ...


2

It sounds like you are trying to compress data, and then recover the same data later. The most common tool for this task is an autoencoder. This model accepts data as input, and then learns to compress it and decompress it to produce something as close as possible to the original data. By making the middle layer of an autoencoder narrower, you can make the ...


2

Simply said, there is no specific "meaning" to the features generated. They are simply features that are fitted through math and calculus, and nobody knows what they represent exactly, and will never knows. However we can run PCA (Principal Component Analysis) to see which feature is the most "important" of all, aka which feature affects the most in the ...


2

Perhaps you are getting checkerboard artifacts Explained here, solutions involve changing the kernel and stride size to prevent them from being not divisible. Besides that, a solution could be to apply Gaussian smoothing to minimize the artifacts. For example, using Gaussian smoothing in OpenCV with your image results in import cv2 img = cv2.imread('s.png') #...


2

In the source code, the author defines sd by sd = 0.5 * tf.layers.dense(x, units=n_latent) which means that $\operatorname{sd}\in \mathbb{R}^n$. In particular, the support over sd includes negative numbers, which is something we want to avoid. Since standard deviations are always nonnegative, we can exponentiate to get us in the correct domain. ...


2

On page 5 of the VAE paper, it's clearly stated We let $p_{\boldsymbol{\theta}}(\mathbf{x} \mid \mathbf{z})$ be a multivariate Gaussian (in case of real-valued data) or Bernoulli (in case of binary data) whose distribution parameters are computed from $\mathbf{z}$ with a MLP (a fully-connected neural network with a single hidden layer, see appendix $\mathrm{...


1

An autoencoder learns to compress data, and then to decompress it again, recovering the original data. It does this by learning a mapping from the original feature space to a lower-dimensional space, and then another mapping back. This is indeed, like PCA. The same technique is used to compress JPEG images to transmit them over the web. Autoencoders can ...


1

I've worked on the BRATS dataset and I can verify that this is pretty much standard process. Besides throwing the totally blank images, I also throw away the images in the beginning and ending of the sequence that show the tip of the scull and the base of the neck. Generally when dealing with MRIs, I do this with a script (think of is as a preprocessing ...


1

Whether a discrete or continuous class, you can model it the same. Denote the encoder $q$ and the decoder $p$. Recall the variational autoencoder's goal is to minimize the $KL$ divergence between $q$ and $p$'s posterior. i.e. $\min_{\theta, \phi} \ KL(q(z|x;\theta) || p(z|x; \phi))$ where $\theta$ and $\phi$ parameterize the encoder and decoder respectively. ...


1

There is no right answer to this. Finding the right loss function is a tough and difficult problem. So your goal as the architect is to try to find one that best suits your needs. So lets think about your needs. You mention that you dont want lighting shifts to cause large error, so ill take a leap and assume you care more about the shapes and style of ...


1

Other replies are commenting on the skip connections for a U-Net. I believe you want to exclude these skip connections from your auto-encoder. You say you want to use the auto-encoder for unsupervised pretraining, for which you want to pass the data through a bottle neck, so adding skip connections would work against you if you want to use the encoder for a ...


1

can I use a different architecture for the decoder, or will this introduce weird artifacts? If you are using U-net -like architecture with skip connection from corresponding encoding to decoding layer outputs of corresponding layers should have the same spatial resolution. There is no other commonly recognized limitations on decoder architecture for ...


1

If your interest is positional information, encode it! This could include learning an embedding for each position and leveraging that in your model. You could also use an approach to hard-encode rather than learn it (kinda like adding sinusoids in the transformer paper Attention is All You Need an example of a paper that encodes the 2D positional info: ...


Only top voted, non community-wiki answers of a minimum length are eligible