How can I reduce the loss? Why do I have the high loss and why do I have the gradient?

Question

I want to classify some images (there are about 200.000 images) with a CNN. But I get a very high loss, see figures:

Loss over the hole training run

Loss for each epoch

It's confused me, that there is a high gradient by the 15 batch of each epoch

I try some option in the history like editing the learning rate and add weights by initialization the network, but I dont unterstand what's wrong. In the first step I train my network with 2000 images, but it's possible to train it with more images.

The Question

How can I reduce the loss and why do I have the high loss? Rather, why do I have the high negativ gradient in the middle of each epoch (see in the second plot)?

Thank you for your help in foward :)

Some words to the dataset:

The dataset is a set of images and they looks very similar. There are two classes: the "good" images and the "error" images. For us - the humans - we will say, that's both images classes looks right, but a normal camera has not enough intelligence. That's the motivation for the projekt.

Now, you find here the code of my neural network:

The training method

model = net.Netz()
optimizer = optim.SGD(model.parameters(), lr= 0.0001, momentum = 0.8)

def trainM(epoch):
    model.train()
    for batch_id, (data, target) in enumerate(net.train_data):
        #data = data.cuda()
        #target = target.cuda()
        
        target = torch.LongTensor(target[64*batch_id:64*(batch_id+1)])
        #data   = torch.Tensor(data[64*batch_id:64*(batch_id+1)])
        data = Variable(data)
        target = Variable(target)
        optimizer.zero_grad()

        out = model(data)
        criterion = F.nll_loss

        loss = criterion(out,target)
        loss.backward()
        torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1)
        optimizer.step()
        print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(epoch,batch_id*len(data), len(net.train_data)*64, 100*batch_id/len(net.train_data), loss.item()))

for item in range(1,10):
    trainM(item)

The CNN

class Netz(nn.Module):
    def __init__(self):
        super(Netz, self).__init__()
        self.conv1 = nn.Conv2d(1, 10, kernel_size=5)
        torch.nn.init.xavier_uniform(self.conv1.weight)
        self.conv2 = nn.Conv2d(10,20, kernel_size = 5)
        torch.nn.init.xavier_uniform(self.conv2.weight)
        self.conv_dropout = nn.Dropout2d()
        self.fc1 = nn.Linear(1050,60)
        self.fc2 = nn.Linear(60,2)
        self.fce = nn.Linear(20,1)
    
    def forward(self,x):
        x = self.conv1(x)
        x = F.max_pool2d(x, 2)
        x = F.relu(x)
        x = self.conv2(x)
        x = self.conv_dropout(x)
        x = F.max_pool2d(x,2)
        x = F.relu(x)
        #x = x.view(-1,320)
        x = x.reshape(x.shape[0], x.shape[1], -1)
        x = F.relu(self.fc1(x))
        x = self.fc2(x)
        x = self.fce(x.permute(0,2,1)).squeeze(-1)
        return F.log_softmax(x, -1)

The data prep method

def dataPrep(list_of_data, data_path, category, quantity):
    global train_data
    global target_list
    train_data_list = []
    mean = [0.0028]
    std = [1.0001]
    
    transform = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize(torch.Tensor(mean), torch.Tensor(std))
        ])
    
    len_data = len(train_data)
    for item in list_of_data:
        f = random.choice(list_of_data)
        list_of_data.remove(f)
        try:
            img = Image.open(data_path +f)
        except:
            continue
        img_crop = img.crop((310,60,425,240))
        img_tensor = transform(img_crop)
        train_data_list.append(img_tensor)

        if category == True:
            target = 1
        else:
            target = 0
        target_list.append(target)
        
        if len(train_data_list) >=64:
            train_data.append((torch.stack(train_data_list), target_list))
            train_data_list = []
            
        if (len_data*64 + quantity) <= len(train_data)*64:
            break   
    return list_of_data

Answer 1

There is something called Dead Neuron Problem in Relu Activation function which means if your weights becomes too small there will a time when your model will stop learning and there will be no updates in parameters at all. Maybe that's problem in your case.

Answer 2

Your issue with high loss and a large negative gradient in the middle of each epoch could be due to a few factors. Let's explore potential causes and strategies to mitigate this:

1. Learning Rate & Gradient Clipping: Problem: A very high learning rate or inadequate gradient clipping can cause large fluctuations in gradients, especially if your dataset is small or has imbalanced distributions between the two classes. A large negative gradient might indicate that the model is overshooting the optimal direction during backpropagation. Solution: You can try lowering the learning rate further. You're already using a small learning rate (0.0001), but lowering it slightly more (e.g., 1e-5 or 5e-5) might stabilize the training. Also, try adjusting the max_norm value in torch.nn.utils.clip_grad_norm_. Lower it (e.g., from 1 to 0.5) to see if it helps reduce the gradient spikes.
2. Batch Size: Problem: A batch size of 64 may cause unstable gradients if the data is noisy or highly similar. Batch size impacts the noise level in gradient updates. Solution: Experiment with smaller batch sizes (e.g., 32 or 16). This may lead to more stable gradient updates. Smaller batches also allow the network to see more variability in each epoch.
3. Class Imbalance or Similarity: Problem: If the two classes are extremely similar and there's imbalance in the dataset (e.g., more "good" images than "error" images), the model might struggle to distinguish between them, leading to loss spikes. Solution: Ensure that your dataset is balanced. You can also try data augmentation to increase the variability between the classes, which might help the model generalize better. Use Focal Loss: Instead of standard nll_loss, try using Focal Loss to penalize the model less for well-classified examples and more for hard-to-classify ones. This could improve learning for the similar images.
4. Normalization and Data Preprocessing: Problem: The normalization you applied (mean=[0.0028] and std=[1.0001]) is quite unusual. If the images are not centered and normalized appropriately, this can destabilize training. Solution: Recheck your normalization values. Typically, the mean and standard deviation should be calculated from the training dataset. Consider using mean=0.5, std=0.5 as a simple initial setup, or compute actual values from the dataset.
5. Weight Initialization: Problem: If your weight initialization is too aggressive, it can cause sharp gradients early in the training process. Solution: You’re using torch.nn.init.xavier_uniform, which is generally good. However, try switching to He initialization (torch.nn.init.kaiming_uniform_), especially since you are using ReLU activations, which tend to work better with He initialization.
6. Overfitting: Problem: If you have too few training examples or the network is too complex, the model might overfit quickly, causing large gradient spikes as it tries to memorize the dataset. Solution: You can introduce more regularization (increase Dropout2d rate) or add L2 weight decay in your optimizer (weight_decay parameter).
7. Data Shuffle: Problem: If your dataset is not shuffled correctly, it may expose the model to batches of data that are very similar, causing spikes in loss. Solution: Ensure that your DataLoader is shuffling the data at every epoch. Plot Analysis If you see a large negative gradient in the middle of each epoch:

Check if your data loader shuffles the data between epochs. This might explain why the gradients behave similarly at the same point each epoch. Plot the learning rate over time to ensure it’s not oscillating or changing too abruptly mid-epoch.

for reference make changes in training code like this one

Consider lowering the learning rate further

optimizer = optim.SGD(model.parameters(), lr=1e-5, momentum=0.8, weight_decay=1e-4)

def trainM(epoch):
    model.train()
    for batch_id, (data, target) in enumerate(net.train_data):
        # Shuffle data if not already done
        data, target = data.cuda(), target.cuda()
        
        optimizer.zero_grad()
        
        out = model(data)
        criterion = F.nll_loss

        loss = criterion(out, target)
        loss.backward()

        # Try lowering gradient clipping norm value
        torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=0.5)

        optimizer.step()

        if batch_id % 15 == 0:  # Log every 15 batches to track potential issues
            print(f'Epoch: {epoch}, Batch: {batch_id}, Loss: {loss.item()}')