How can I reduce the loss? Why do I have the high loss and why do I have the gradient?

Question

I want to classify some images (there are about 200.000 images) with a CNN. But I get a very high loss, see figures:

Loss over the hole training run

Loss for each epoch

It's confused me, that there is a high gradient by the 15 batch of each epoch

I try some option in the history like editing the learning rate and add weights by initialization the network, but I dont unterstand what's wrong. In the first step I train my network with 2000 images, but it's possible to train it with more images.

The Question

How can I reduce the loss and why do I have the high loss? Rather, why do I have the high negativ gradient in the middle of each epoch (see in the second plot)?

Thank you for your help in foward :)

Some words to the dataset:

The dataset is a set of images and they looks very similar. There are two classes: the "good" images and the "error" images. For us - the humans - we will say, that's both images classes looks right, but a normal camera has not enough intelligence. That's the motivation for the projekt.

Now, you find here the code of my neural network:

The training method

model = net.Netz()
optimizer = optim.SGD(model.parameters(), lr= 0.0001, momentum = 0.8)

def trainM(epoch):
    model.train()
    for batch_id, (data, target) in enumerate(net.train_data):
        #data = data.cuda()
        #target = target.cuda()
        
        target = torch.LongTensor(target[64*batch_id:64*(batch_id+1)])
        #data   = torch.Tensor(data[64*batch_id:64*(batch_id+1)])
        data = Variable(data)
        target = Variable(target)
        optimizer.zero_grad()

        out = model(data)
        criterion = F.nll_loss

        loss = criterion(out,target)
        loss.backward()
        torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1)
        optimizer.step()
        print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(epoch,batch_id*len(data), len(net.train_data)*64, 100*batch_id/len(net.train_data), loss.item()))

for item in range(1,10):
    trainM(item)

The CNN

class Netz(nn.Module):
    def __init__(self):
        super(Netz, self).__init__()
        self.conv1 = nn.Conv2d(1, 10, kernel_size=5)
        torch.nn.init.xavier_uniform(self.conv1.weight)
        self.conv2 = nn.Conv2d(10,20, kernel_size = 5)
        torch.nn.init.xavier_uniform(self.conv2.weight)
        self.conv_dropout = nn.Dropout2d()
        self.fc1 = nn.Linear(1050,60)
        self.fc2 = nn.Linear(60,2)
        self.fce = nn.Linear(20,1)
    
    def forward(self,x):
        x = self.conv1(x)
        x = F.max_pool2d(x, 2)
        x = F.relu(x)
        x = self.conv2(x)
        x = self.conv_dropout(x)
        x = F.max_pool2d(x,2)
        x = F.relu(x)
        #x = x.view(-1,320)
        x = x.reshape(x.shape[0], x.shape[1], -1)
        x = F.relu(self.fc1(x))
        x = self.fc2(x)
        x = self.fce(x.permute(0,2,1)).squeeze(-1)
        return F.log_softmax(x, -1)

The data prep method

def dataPrep(list_of_data, data_path, category, quantity):
    global train_data
    global target_list
    train_data_list = []
    mean = [0.0028]
    std = [1.0001]
    
    transform = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize(torch.Tensor(mean), torch.Tensor(std))
        ])
    
    len_data = len(train_data)
    for item in list_of_data:
        f = random.choice(list_of_data)
        list_of_data.remove(f)
        try:
            img = Image.open(data_path +f)
        except:
            continue
        img_crop = img.crop((310,60,425,240))
        img_tensor = transform(img_crop)
        train_data_list.append(img_tensor)

        if category == True:
            target = 1
        else:
            target = 0
        target_list.append(target)
        
        if len(train_data_list) >=64:
            train_data.append((torch.stack(train_data_list), target_list))
            train_data_list = []
            
        if (len_data*64 + quantity) <= len(train_data)*64:
            break   
    return list_of_data

Answer 1

There is something called Dead Neuron Problem in Relu Activation function which means if your weights becomes too small there will a time when your model will stop learning and there will be no updates in parameters at all. Maybe that's problem in your case.