# Test accuracy decreases during my train process

I want to train a neural network model with the arcface loss function and try to combine it with domain adaption. But when the training process continues, I find the test accuracy first increases and then decreases, the model cannot reach convergence. I chose the office31 dataset, and the feature_extractor was resnet50.

I want to know if it is caused by my code, or by my loss function

The arcface function was set as

def Arc_pred(cosine, s=64.0, m=0.1):
cosine = cosine / s
thea = torch.acos(cosine)
top = torch.exp(torch.cos(thea + m) * s)
_top = torch.exp(torch.cos(thea) * s)
bottom = torch.sum(torch.exp(cosine * s), dim=1).view(-1, 1)
divide = (top / (bottom - _top + top)) + 1e-10
return divide


and my total loss function was set as

total_loss = 0.1*target_entropy_loss + label_loss + arc_loss + discriminator_loss


In that, the target_entropy_loss tries to make the decision boundary cross the sparsest sample area，label_loss was the classification loss, discriminator_loss was a domain adaptation loss function.

I tried to set a learning rate schedule for my experiment, it seems it did not work. So, could it be caused by my loss function?

• Hello. From the plot you're showing us, it's not clear what is the highest test accuracy reached after it drops. As suggested in the answer below, you could try early stopping, but if the top test accuracy is too low, you may try something else first. Could you please also tell us how long have you been training the model for, at which epoch the test accuracy drops and how large are your training and test datasets.
– nbro
Nov 25 '21 at 1:05
• I tried to train the network by 10000 iterations,the highest test accuracy reached at about 91%.Besides overfitting,I want to know whether the SGD optimizer I chosed for the GAN part also could cause the decrease. Nov 27 '21 at 1:39