0
$\begingroup$

I am training a WGAN-GP network based on the following paper, though I am using a different dataset. Now, for the first ~ 60-70 epochs, my network trained really well, which I could see in the loss going down, but I also made sure to regularly check the quality of the images.

Unfortunately, what I am seeing now (for the last $20$ epochs) is that the generator is getting worse and worse, the images don't look that good anymore. I save checkpoints every epochs, so in principle, I could stop training and get myself a state of the network from where it was still performing quite okay.

However, my question would be: How can I improve the training of the GAN? Would you decrease the learning rate?

I use a batch size of 124 and a learning rate of 1e-3. Maybe I could/should continue training (with a checkpoint that was still quite okay) with a learning rate of 5e-4?

Any other hints would be appreciated!

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

This is from my own experience with (Vanilla) GANs, so it might not translate exactly to your application, but maybe it gives some orientation.

  • your learning rate seems quite high. I've quite frequently found that 1e-5 is a good value for me. The training might take longer but will probably be more stable.
  • have you tried using dropout? It's a good regularisation mechanism that can prevent from overfitting and also make training more stable. I've made good experience with a dropout rate of ~0.4 and keeping it in both training and testing stages.
  • what is your discriminator vs generator update ratio? In my experiments, ~5 discriminator updates per generator update has often shown to improve things.
  • how many training samples do you have? If they are limited, it might be helpful to add a bit of noise on them. This manipulates the data a bit, but might make training more stable.
  • I have often not seen a major change when varying the batch size.
  • what are your network architectures? I have the feeling that small networks are often sufficient...

This is entirely based on my own experience, but maybe there are some interesting directions for you to explore. Happy to hear back from you about the success of these tests.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Hi Mafu, thanks! (i) You're right, the LR of $1e-3$ seems to high.. Maybe I will restart training from scratch and see what it does. (ii) Interesting, I didn't know one could also use Dropout with GANs.. (iii) My critic (as for WGANs, we don't strictly have a discriminator) updates itself $5$-times before a generator update. (iv) I have enough training samles (more than $200$k images). (v) My network architectures are heavy, I won't deny that.. I oriented myself at this: github.com/aladdinpersson/Machine-Learning-Collection/blob/… $\endgroup$
    – Anonymous5638
    Commented Mar 16, 2021 at 8:17

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .