How can I reduce the loss? Why do I have the high loss and why do I have the gradient?

I want to classify some images (there are about 200.000 images) with a CNN. But I get a very high loss, see figures:

Loss over the hole training run

Loss for each epoch

It's confused me, that there is a high gradient by the 15 batch of each epoch

I try some option in the history like editing the learning rate and add weights by initialization the network, but I dont unterstand what's wrong. In the first step I train my network with 2000 images, but it's possible to train it with more images.

The Question

How can I reduce the loss and why do I have the high loss? Rather, why do I have the high negativ gradient in the middle of each epoch (see in the second plot)?

Thank you for your help in foward :)

Some words to the dataset:

The dataset is a set of images and they looks very similar. There are two classes: the "good" images and the "error" images. For us - the humans - we will say, that's both images classes looks right, but a normal camera has not enough intelligence. That's the motivation for the projekt.

Now, you find here the code of my neural network:

The training method

model = net.Netz()
optimizer = optim.SGD(model.parameters(), lr= 0.0001, momentum = 0.8)

def trainM(epoch):
model.train()
for batch_id, (data, target) in enumerate(net.train_data):
#data = data.cuda()
#target = target.cuda()

target = torch.LongTensor(target[64*batch_id:64*(batch_id+1)])
#data   = torch.Tensor(data[64*batch_id:64*(batch_id+1)])
data = Variable(data)
target = Variable(target)

out = model(data)
criterion = F.nll_loss

loss = criterion(out,target)
loss.backward()
optimizer.step()
print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(epoch,batch_id*len(data), len(net.train_data)*64, 100*batch_id/len(net.train_data), loss.item()))

for item in range(1,10):
trainM(item)


The CNN

class Netz(nn.Module):
def __init__(self):
super(Netz, self).__init__()
self.conv1 = nn.Conv2d(1, 10, kernel_size=5)
torch.nn.init.xavier_uniform(self.conv1.weight)
self.conv2 = nn.Conv2d(10,20, kernel_size = 5)
torch.nn.init.xavier_uniform(self.conv2.weight)
self.conv_dropout = nn.Dropout2d()
self.fc1 = nn.Linear(1050,60)
self.fc2 = nn.Linear(60,2)
self.fce = nn.Linear(20,1)

def forward(self,x):
x = self.conv1(x)
x = F.max_pool2d(x, 2)
x = F.relu(x)
x = self.conv2(x)
x = self.conv_dropout(x)
x = F.max_pool2d(x,2)
x = F.relu(x)
#x = x.view(-1,320)
x = x.reshape(x.shape[0], x.shape[1], -1)
x = F.relu(self.fc1(x))
x = self.fc2(x)
x = self.fce(x.permute(0,2,1)).squeeze(-1)
return F.log_softmax(x, -1)


The data prep method

def dataPrep(list_of_data, data_path, category, quantity):
global train_data
global target_list
train_data_list = []
mean = [0.0028]
std = [1.0001]

transform = transforms.Compose([
transforms.ToTensor(),
transforms.Normalize(torch.Tensor(mean), torch.Tensor(std))
])

len_data = len(train_data)
for item in list_of_data:
f = random.choice(list_of_data)
list_of_data.remove(f)
try:
img = Image.open(data_path +f)
except:
continue
img_crop = img.crop((310,60,425,240))
img_tensor = transform(img_crop)
train_data_list.append(img_tensor)

if category == True:
target = 1
else:
target = 0
target_list.append(target)

if len(train_data_list) >=64:
train_data.append((torch.stack(train_data_list), target_list))
train_data_list = []

if (len_data*64 + quantity) <= len(train_data)*64:
break
return list_of_data

• Have you tried increasing the number of neurons or layers ? And Try to use Adam Optimizer instead of SGD !
– JOGI
Apr 24, 2022 at 13:16
• Yes I tried it, but the result is the same :( Apr 24, 2022 at 20:15
• There might be some bug in the code. From a first glimpse I don't really understand what's going on after your conv-layers. Specifically, this line preps the tensors for the classification head: x.reshape(x.shape[0], x.shape[1], -1) so you get a tensor of shape batchsize x img-height x features. By doing that you effectively run your classification head on the img rows separately, right? So they have very little info about the full input image. Is there a reason why you don't reshape the thing by x.reshape(x.shape[0], -1) and use a standard MLP as the classification head? May 8, 2022 at 20:55