# How to stabilize the training of a Conv-Siamese Neural Network if the results after different trainings vary relatively strongly?

I am training a neural network using MSE and ADAM optimizer. More precisely, a siamese architecture with a convolutional encoder and euclidean distance on top. I am using MSE because I have different similarity values between each training pair and results are better than using BCE with strict 0 and 1 labeling. As a stop criterion for the training, I use the criterion that there was no improvement of the training loss after 500 iterations and then choose the NN model with the lowest loss value for evaluation. I do not using an extra validation set because it takes some time and will slow down training and I have a few-shot learning problem (I am not doing classification, its instance based learning). I also set 2500 training iterations as maximum of training epochs, as results with more iteration (around 4000) seems to be more worse. Sometimes it stops at 2500, other times it stops at around 1600.

Here are some values of the training loss:

Epoch: 0 - Loss: 0.21525
Epoch: 10 - Loss: 0.14386
Epoch: 20 - Loss: 0.12428
Epoch: 30 - Loss: 0.11221
Epoch: 40 - Loss: 0.07352
Epoch: 50 - Loss: 0.09534
Epoch: 60 - Loss: 0.08777
Epoch: 70 - Loss: 0.08288
Epoch: 80 - Loss: 0.05025
Epoch: 90 - Loss: 0.04680
Epoch: 100 - Loss: 0.06165
Epoch: 110 - Loss: 0.04287
Epoch: 120 - Loss: 0.05606
Epoch: 130 - Loss: 0.04665
Epoch: 140 - Loss: 0.04451
Epoch: 150 - Loss: 0.03439
Epoch: 160 - Loss: 0.06105
Epoch: 170 - Loss: 0.04814
Epoch: 180 - Loss: 0.06930
Epoch: 190 - Loss: 0.05361
Epoch: 200 - Loss: 0.06833

...

Epoch: 2300 - Loss: 0.02062
Epoch: 2310 - Loss: 0.02665
Epoch: 2320 - Loss: 0.03478
Epoch: 2330 - Loss: 0.06372
Epoch: 2340 - Loss: 0.02842
Epoch: 2350 - Loss: 0.02382
Epoch: 2360 - Loss: 0.01602
Epoch: 2370 - Loss: 0.02578
Epoch: 2380 - Loss: 0.02289
Epoch: 2390 - Loss: 0.01732
Epoch: 2400 - Loss: 0.03164
Epoch: 2410 - Loss: 0.02156
Epoch: 2420 - Loss: 0.05448
Epoch: 2430 - Loss: 0.01680
Epoch: 2440 - Loss: 0.02365
Epoch: 2450 - Loss: 0.01704
Epoch: 2460 - Loss: 0.04040
Epoch: 2470 - Loss: 0.01970
Epoch: 2480 - Loss: 0.01718
Epoch: 2490 - Loss: 0.01838


The issue is, that my results vary considerably when training it multiple times with the same procedure. But in more than a half of my trainings the models I get performing extremly well and provide great results.

What are typical ways to stabalize the training procedure to get more consistent results?