8

I see no reason why decaying learning rates should create the kinds of jumps in losses that you are observing. It should "slow down" how quickly you "move", which in the case of a loss that otherwise consistently shrinks really should, at worst, just lead to a plateau in your losses (rather than those jumps). The first thing I observe in your code is that ...


6

It is important to think about what sort of patterns in the data are being represented. Suppose that you have a dataset of greyscale images, such that every image is a uniform intensity. As a human brain you'd realise that every element in this dataset can be described in terms of a single numeric parameter, which is that intensity value. This is something ...


4

Let's assume the probability distributions are Gaussian (or normal) distributions. In other words, in the Bayes' rule \begin{align} p(z|x)=\frac{p(x|z)p(z)}{p(x)} \tag{1}\label{1} \end{align} The posterior $p(z|x)$, the likelihood $p(x|z)$, the prior $p(z)$ and the evidence (or marginal) $p(x)$ are Gaussian distributions. You can assume this because ...


4

In fact, autoencoders are used for generative tasks. Have a look at Tutorial on Variational Autoencoders (VAEs). The coolest thing about VAE is that abstract features can be easily amplified or suppressed based on extracted vectors from the latent space. Let's imagine a model trained on MNIST to generate digits. If you take two images of the same digit which ...


4

Auto-encoders are widely used and maybe even more used than GANs (in fact, auto-encoders are older than GANs, although the main general idea behind GANs is quite old). For example, auto-encoders are used in World Models, for drug design (e.g. see this paper) and many other tasks that involve data compression or generation. So, if we train autoencoders, for ...


3

The key is: VAE usually use a small latent dimension, the information of input is so hard to pass through this bottleneck, meanwhile it tries to minimize the loss with the batch of input data, you should know the result -- VAE can only have a mean and blurry output. If you increase the bandwidth of the bottleneck, i.e. the size of latent vector, VAE can get ...


3

In essence, Variational Autoencoders learn an "explicit" distribution of the data by trying to fit the data via a multi-dimensional Gaussian/Normal distribution. However, Generative Adversarial Networks learn an "implicit" distribution of data meaning that you cannot directly sample them. Also, due to the deterministic nature of neural networks GANs tend to ...


3

With Generative Adversarial Networks, all the generator cares about is fooling the discriminator. There's no requirement to be clever, or exhaustive, or make efficient use of the input space. As long as the discriminator returns "real" (vs. "fake") the generator "wins". The hope is that as the generator and discriminator are trained simultaneously, each ...


3

PCA is a linear method that creates a transformation that is capable of changing the vectors projections (changing axis) Since PCA looks for the direction of maximum variance it usually have high discriminativity BUT it does not guaranteed that the direction of most variance is the direction of most discriminativity. LDA is a linear method that creates a ...


3

A use case of autoencoders (in particular, of the decoder or generative model of the autoencoder) is to denoise the input. This type of autoencoders, called denoising autoencoders, take a partially corrupted input and they attempt to reconstruct the corresponding uncorrupted input. There are several applications of this model. For example, if you had a ...


3

The number of dimensions is a hyperparameter of your model, and you should do a hyperparameter search, like with any other parameters. There's also a tradeoff between dimension and training speed, so it should be small enough to be trainable in a reasonable time.


3

Theory Encoder In general, an Encoder is a mapping $f : X \rightarrow Y $ with $X$ Input Space and $Y$ Code Space In case of Neural Networks, it is a Generative Model hence a function which is able to compute a Representation out of some input (like GAN) The point is: how would you train such an encoder network ? The general answer is: it depends on ...


3

Both are specific types of a broad class of what are now usually called Evolutionary Algorithms. There's no single definition of what makes an Evolutionary Algorithm, but it's generally construed to be very broad. A rough description might be any search method that starts with a population of one or more stochastically generated candidate solutions to a ...


2

I can not seem to grasp which specs make an encoder better than another one In general, in unsupervised settings, we want to learn the probability distribution of the data p(x) by some latent variables that explain the variations observed in the training set. The autoencoder family (Variational, Denoising, Contrastive, Sparse) try to approximate p(x) ...


2

The paper refers to layers and sub-layers, and clearly indicates that one layer includes all three sub-layers, so when they say they train all three layers simultaneously, they are talking about the three autoencoder layers, not the sub-layers. This also agrees with the fact that only the autoencoder layer has tunable parameters. The other two layers use ...


2

I believe that the idea is to have a similar ratio of fraud/"normal transaction" as to the ones that bank encounter on real life. If you balance it you will probably have a lot of false positive once you apply your solution to real world's data and, if that may be fine for you to play with, it's not what a bank would like as they can't block too much of the ...


2

I have not worked on this but I think I can give you a theoretical perspective of using VAE's. Regression is a Supervised Learning task and is basically a mapping from Input to Output where the Neural Net will approximate the function $f(input) = output$. VAE's on the other hand are good for finding how a latent variable affects the output. For example, if ...


2

Is this a good approach? Will I have a lot of trouble with different backgrounds? A lot will depend on the nature of the backgrounds you have, and how well they encode/decode by themselves without the object in frame. My gut feeling is that your system will have poor performance compared to a properly trained classifier, as the autoencoder will naturally ...


2

It sounds like you are trying to compress data, and then recover the same data later. The most common tool for this task is an autoencoder. This model accepts data as input, and then learns to compress it and decompress it to produce something as close as possible to the original data. By making the middle layer of an autoencoder narrower, you can make the ...


2

Simply said, there is no specific "meaning" to the features generated. They are simply features that are fitted through math and calculus, and nobody knows what they represent exactly, and will never knows. However we can run PCA (Principal Component Analysis) to see which feature is the most "important" of all, aka which feature affects the most in the ...


2

In the source code, the author defines sd by sd = 0.5 * tf.layers.dense(x, units=n_latent) which means that $\operatorname{sd}\in \mathbb{R}^n$. In particular, the support over sd includes negative numbers, which is something we want to avoid. Since standard deviations are always nonnegative, we can exponentiate to get us in the correct domain. ...


1

An autoencoder learns to compress data, and then to decompress it again, recovering the original data. It does this by learning a mapping from the original feature space to a lower-dimensional space, and then another mapping back. This is indeed, like PCA. The same technique is used to compress JPEG images to transmit them over the web. Autoencoders can ...


1

I've worked on the BRATS dataset and I can verify that this is pretty much standard process. Besides throwing the totally blank images, I also throw away the images in the beginning and ending of the sequence that show the tip of the scull and the base of the neck. Generally when dealing with MRIs, I do this with a script (think of is as a preprocessing ...


1

Whether a discrete or continuous class you can model it the same. Denote the encoder $q$ and the decoder $p$. Recall the variational autoencoder's goal is to minimize the $KL$ divergence between $q$ and $p$'s posterior. i.e. $\min_{\theta, \phi} \ KL(q(z|x;\theta) || p(z|x; \phi))$ where $\theta$ and $\phi$ parameterize the encoder and decoder ...


1

There is no right answer to this. Finding the right loss function is a tough and difficult problem. So your goal as the architect is to try to find one that best suits your needs. So lets think about your needs. You mention that you dont want lighting shifts to cause large error, so ill take a leap and assume you care more about the shapes and style of ...


1

Other replies are commenting on the skip connections for a U-Net. I believe you want to exclude these skip connections from your auto-encoder. You say you want to use the auto-encoder for unsupervised pretraining, for which you want to pass the data through a bottle neck, so adding skip connections would work against you if you want to use the encoder for a ...


1

can I use a different architecture for the decoder, or will this introduce weird artifacts? If you are using U-net -like architecture with skip connection from corresponding encoding to decoding layer outputs of corresponding layers should have the same spatial resolution. There is no other commonly recognized limitations on decoder architecture for ...


1

If your interest is positional information, encode it! This could include learning an embedding for each position and leveraging that in your model. You could also use an approach to hard-encode rather than learn it (kinda like adding sinusoids in the transformer paper Attention is All You Need an example of a paper that encodes the 2D positional info: ...


1

The main difference between a variational auto-encoder (VAE) and an auto-encoder is that the VAE is a generative and statistical model while an auto-encoder (AE) is just a data compressor and decompressor (it is just a function approximator), so an AE is not a statistical model. They both have an encoder and a decoder and they both convert the inputs to a ...


1

It was not about the model, but about ImageDataGenerator. I found the answer here: https://stackoverflow.com/a/51673998/493080


Only top voted, non community-wiki answers of a minimum length are eligible