Test accuracy decreases during my train process

Question

I want to train a neural network model with the arcface loss function and try to combine it with domain adaption. But when the training process continues, I find the test accuracy first increases and then decreases, the model cannot reach convergence. I chose the office31 dataset, and the feature_extractor was resnet50.

I want to know if it is caused by my code, or by my loss function

The arcface function was set as

def Arc_pred(cosine, s=64.0, m=0.1):
    cosine = cosine / s
    thea = torch.acos(cosine)
    top = torch.exp(torch.cos(thea + m) * s)
    _top = torch.exp(torch.cos(thea) * s)
    bottom = torch.sum(torch.exp(cosine * s), dim=1).view(-1, 1)
    divide = (top / (bottom - _top + top)) + 1e-10
    return divide

and my total loss function was set as

total_loss = 0.1*target_entropy_loss + label_loss + arc_loss + discriminator_loss

In that, the target_entropy_loss tries to make the decision boundary cross the sparsest sample area，label_loss was the classification loss, discriminator_loss was a domain adaptation loss function.

I tried to set a learning rate schedule for my experiment, it seems it did not work. So, could it be caused by my loss function?

Answer 1

This looks like overfitting. You can try stop training earlier by using a validation dataset to prevent this, or you can try other regularization effects such as weight-decay, dropout etc.