enter image description here

I want to train a neural network model with the arcface loss function and try to combine it with domain adaption. But when the training process continues, I find the test accuracy first increases and then decreases, the model cannot reach convergence. I chose the office31 dataset, and the feature_extractor was resnet50.

I want to know if it is caused by my code, or by my loss function

The arcface function was set as

def Arc_pred(cosine, s=64.0, m=0.1):
    cosine = cosine / s
    thea = torch.acos(cosine)
    top = torch.exp(torch.cos(thea + m) * s)
    _top = torch.exp(torch.cos(thea) * s)
    bottom = torch.sum(torch.exp(cosine * s), dim=1).view(-1, 1)
    divide = (top / (bottom - _top + top)) + 1e-10
    return divide

and my total loss function was set as

total_loss = 0.1*target_entropy_loss + label_loss + arc_loss + discriminator_loss

In that, the target_entropy_loss tries to make the decision boundary cross the sparsest sample area,label_loss was the classification loss, discriminator_loss was a domain adaptation loss function.

I tried to set a learning rate schedule for my experiment, it seems it did not work. So, could it be caused by my loss function?

  • $\begingroup$ Hello. From the plot you're showing us, it's not clear what is the highest test accuracy reached after it drops. As suggested in the answer below, you could try early stopping, but if the top test accuracy is too low, you may try something else first. Could you please also tell us how long have you been training the model for, at which epoch the test accuracy drops and how large are your training and test datasets. $\endgroup$
    – nbro
    Nov 25 at 1:05
  • $\begingroup$ I tried to train the network by 10000 iterations,the highest test accuracy reached at about 91%.Besides overfitting,I want to know whether the SGD optimizer I chosed for the GAN part also could cause the decrease. $\endgroup$
    – klayoe
    Nov 27 at 1:39

This looks like overfitting. You can try stop training earlier by using a validation dataset to prevent this, or you can try other regularization effects such as weight-decay, dropout etc.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.