I am fairly new to deep learning and I have been testing out several architectures for the segmentation task of clouds in satellite imagery. I am using a simple Unet as my benchmark, Unet++, Efficient Net, Unet with a Resnet Backbone. And for the sake of experimenting, I branched out to transfer learning using VGG16 and ResNet34 (by removing the classifier and adding a decoder). My problem is the learning (train and validation) and the loss curves, they are very similar for all architectures and showing a spiky behavior as in the figures bellow. For the loss functions I tried -log (Dice), BCE, DiceBCE, and it's always the spiky behaviour. For the Evaluation Metrics, I am using IoU, Dice coeff, Precision, and Recall. I tried using both the lr=1e-4 and lr=1e-5 as well as increasing the batch size. But nothing seems to work. My dataset consists of 10000 images of size (3, 384, 384). Any suggestions on how I can improve on this? or what could I have possibly done wrong?