I am working with generative adversarial networks (GANs) and one of my aims at the moment is to reproduce samples in two dimensions that are distributed according to a circle (see animation). When using a GAN with small networks (3 layers with 50 neurons each), the results are more stable than with bigger layers (3 layers with 500 neurons each). All other hyperparameters are the same (see details of my implementation below).

I am wondering if anyone has an explanation why this is the case. I could obviously try to tune the other hyperparameters to get good performance, but would be interested in knowing if someone has heuristics about what is needed to change whenever I change the size of the networks.

Thanks in advance!

GAN with smaller layer size reproduces the original samples better

Network/Training parameters

I use PyTorch with the following settings for the GAN:


  • Generator/Discriminator Architecture (all dense layers): 100-50-50-50-2 (small); 100-500-500-500-2 (big)
  • Dropout: p=0.4 for generator (except last layer), p=0 for discriminator
  • Activation functions: LeakyReLU (slope 0.1)


  • Optimizer: Adam
  • Learning Rate: 1e-5 (for both networks)
  • Beta1, Beta2: 0.9, 0.999
  • Batch size: 50
  • $\begingroup$ How many data points did you used? Maybe increasing dataset n times reduces this effect. $\endgroup$ – Enes E Jan 26 at 9:16
  • $\begingroup$ I use 100k data points in total for training. With a batch size of 50, that's 2000 updates for each epoch, and I have 600 epochs. I think 100k is big enough so that increasing it shouldn't change too much. $\endgroup$ – Mafu Jan 26 at 9:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.