0
$\begingroup$

I am training a semi-supervised GAN network using data from multiple subjects. I separated the labeled and unlabeled data based on my subjects, so there is no leakage, while having much more unlabeled data than labeled data. After few epochs training accuracy hits 100% which normally indicates overfitting, however the performance on the validation and test sets keeps increasing for 200-300 epochs. Is this considered overfitting and is there an explanation for this behavior?

Train and validation curves during training.

Improve this question
$\endgroup$
6
  • 1
    $\begingroup$ There is a lot of variables here which can help answer this. The main one: How are you measuring accuracy? For example, if you measure a correct prediction as above 0.5 when positive and below 0.5 when negative, the accuracy could be 100% well before overfitting, as the network still has a lot of room to approach 1 and 0. $\endgroup$
    – Recessive
    Commented Oct 14, 2021 at 3:30
  • $\begingroup$ I am in a multiclass setting with k=12 classes and I use argmax on the output logits, and compare them to real labels using sklearn's accuracy function. I am also computing the loss, for example in one epoch i had 0.4 for validation and 0.01 for training, and later training loss drops to 1e-5 while validation loss oscillates around 1, with no noticable impact on the accuracy score. $\endgroup$
    – Hazar
    Commented Oct 14, 2021 at 11:50
  • $\begingroup$ How does it do on the test set? This does sound like it's overfitting based on the loss. I would triple check you are definitely running the same version of the network that's producing this loss when computing accuracy before continuing, because it does sound quite strange that loss would increase by over 100% with accuracy also increasing at the same time $\endgroup$
    – Recessive
    Commented Oct 15, 2021 at 1:05
  • $\begingroup$ Same on the test set, performances kept increasing even after signs of overfitting on the training set. For example in a recent model, I obtained 100% acc, 0.0016 loss for training, around 93% acc and 0.4 loss for both validation and test sets that came from different subjects. $\endgroup$
    – Hazar
    Commented Oct 16, 2021 at 2:06
  • 1
    $\begingroup$ Can you maybe post a graph of the loss and accuracy for training and validation? That could help see what's going on here more exactly. $\endgroup$
    – Recessive
    Commented Oct 16, 2021 at 11:34

2 Answers 2

Reset to default
1
$\begingroup$

You did a great job at this...

You can use the Tensorflow’s LogSumExp built-in function to avoid numerical problems. This routine prevents over/under flow issues that may occur when LogSumExp encounters very extreme, either positive or negative values.

You have sorted out this:

There need to be Images from the generator. To these ones, the discriminator learns to classify as fake.

Real images with labels. These are image label pairs like in any regular supervised classification problem.

Real images without labels. For those, the classifier only learns that these images are real.

I would recommend you visit this link: Semi-supervised learning with Generative Adversarial Networks (GANs)

Good luck.

Improve this answer
$\endgroup$
1
$\begingroup$

If you ruled out leakage completely read this observation about double decent https://openai.com/blog/deep-double-descent/ This Blogpost from openAI shares the observation that the validation loss can decrease again even after initially increasing (which is typically a sign for the start of overfitting, e.g. in early stopping).

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .