I want to classify some images (there are about 200.000 images) with a CNN. But I get a very high loss, see figures:

Loss over the hole training run

enter image description here

Loss for each epoch

enter image description here

It's confused me, that there is a high gradient by the 15 batch of each epoch

I try some option in the history like editing the learning rate and add weights by initialization the network, but I dont unterstand what's wrong. In the first step I train my network with 2000 images, but it's possible to train it with more images.

The Question

How can I reduce the loss and why do I have the high loss? Rather, why do I have the high negativ gradient in the middle of each epoch (see in the second plot)?

Thank you for your help in foward :)

Some words to the dataset:

The dataset is a set of images and they looks very similar. There are two classes: the "good" images and the "error" images. For us - the humans - we will say, that's both images classes looks right, but a normal camera has not enough intelligence. That's the motivation for the projekt.

Now, you find here the code of my neural network:

The training method

model = net.Netz()
optimizer = optim.SGD(model.parameters(), lr= 0.0001, momentum = 0.8)

def trainM(epoch):
    for batch_id, (data, target) in enumerate(net.train_data):
        #data = data.cuda()
        #target = target.cuda()
        target = torch.LongTensor(target[64*batch_id:64*(batch_id+1)])
        #data   = torch.Tensor(data[64*batch_id:64*(batch_id+1)])
        data = Variable(data)
        target = Variable(target)

        out = model(data)
        criterion = F.nll_loss

        loss = criterion(out,target)
        torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1)
        print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(epoch,batch_id*len(data), len(net.train_data)*64, 100*batch_id/len(net.train_data), loss.item()))

for item in range(1,10):


class Netz(nn.Module):
    def __init__(self):
        super(Netz, self).__init__()
        self.conv1 = nn.Conv2d(1, 10, kernel_size=5)
        self.conv2 = nn.Conv2d(10,20, kernel_size = 5)
        self.conv_dropout = nn.Dropout2d()
        self.fc1 = nn.Linear(1050,60)
        self.fc2 = nn.Linear(60,2)
        self.fce = nn.Linear(20,1)
    def forward(self,x):
        x = self.conv1(x)
        x = F.max_pool2d(x, 2)
        x = F.relu(x)
        x = self.conv2(x)
        x = self.conv_dropout(x)
        x = F.max_pool2d(x,2)
        x = F.relu(x)
        #x = x.view(-1,320)
        x = x.reshape(x.shape[0], x.shape[1], -1)
        x = F.relu(self.fc1(x))
        x = self.fc2(x)
        x = self.fce(x.permute(0,2,1)).squeeze(-1)
        return F.log_softmax(x, -1)

The data prep method

def dataPrep(list_of_data, data_path, category, quantity):
    global train_data
    global target_list
    train_data_list = []
    mean = [0.0028]
    std = [1.0001]
    transform = transforms.Compose([
    transforms.Normalize(torch.Tensor(mean), torch.Tensor(std))
    len_data = len(train_data)
    for item in list_of_data:
        f = random.choice(list_of_data)
            img = Image.open(data_path +f)
        img_crop = img.crop((310,60,425,240))
        img_tensor = transform(img_crop)

        if category == True:
            target = 1
            target = 0
        if len(train_data_list) >=64:
            train_data.append((torch.stack(train_data_list), target_list))
            train_data_list = []
        if (len_data*64 + quantity) <= len(train_data)*64:
    return list_of_data
  • $\begingroup$ Have you tried increasing the number of neurons or layers ? And Try to use Adam Optimizer instead of SGD ! $\endgroup$
    – JOGI
    Apr 24, 2022 at 13:16
  • $\begingroup$ Yes I tried it, but the result is the same :( $\endgroup$ Apr 24, 2022 at 20:15
  • $\begingroup$ There might be some bug in the code. From a first glimpse I don't really understand what's going on after your conv-layers. Specifically, this line preps the tensors for the classification head: x.reshape(x.shape[0], x.shape[1], -1) so you get a tensor of shape batchsize x img-height x features. By doing that you effectively run your classification head on the img rows separately, right? So they have very little info about the full input image. Is there a reason why you don't reshape the thing by x.reshape(x.shape[0], -1) and use a standard MLP as the classification head? $\endgroup$
    – Chillston
    May 8, 2022 at 20:55

1 Answer 1


There is something called Dead Neuron Problem in Relu Activation function which means if your weights becomes too small there will a time when your model will stop learning and there will be no updates in parameters at all. Maybe that's problem in your case.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .