0
$\begingroup$

I am very new to this pytorch and neural networks.I am stuck in training one model since last 1 week. My model paramters are not getting updated after each epoch. Also,list(model.paramteres()[-1].grad) returns None

This is the code I wrote.

  for epoch in range(10):
    running_loss = 0.0
    print(list(net.parameters())[6].grad )
    outputs = net(inputs.float())
    a=torch.max(outputs,1).indices
    a=a.float()
    labels=labels.float()
    print(list(net.parameters())[6].grad )

    loss = criterion(a,labels.squeeze(1))
  
    loss = torch.tensor(loss, requires_grad = True)
    print(list(net.parameters())[6].grad )
    optimizer.zero_grad()
    
    loss.backward()
    
    optimizer.step()
    print(list(net.parameters())[6].grad )        

print('Finished Training')

also, this is the optimizer I have used,

import torch.optim as optim
criterion = nn.BCELoss()
optimizer = optim.Adam(net.parameters(), lr=0.001,weight_decay=0.0001)
from torch.autograd import Variable

Basically, what I did is I tried to train the model on only one batch of batch size=16 in order to see if my weights are getting updated or not.And I found that my loss, and the weights all remain same for 100 iterations.I really can't figure out why is this happening.

I also have another question as well. My dataset contains grayscale images.And I need to perform a binary classification on them. This is the model I have made.It takes as an input tensor of shape (N,1,64,64) where N is the batch_size.

import torch
import torch.nn as nn
import torch.nn.functional as F
class Net(nn.Module):

    def __init__(self):
        super(Net, self).__init__()

        self.conv1 = nn.Conv2d(1, 16, 5)
        self.conv2 = nn.Conv2d(16, 32, 7)

        self.dropout1 = nn.Dropout2d(0.25)
        self.dropout2 = nn.Dropout2d(0.5)
        self.fc1 = nn.Linear(4608,128)  
        self.fc2 = nn.Linear(128,16)
        self.fc3 = nn.Linear(16, 2)


    def forward(self, x):
        # Max pooling over a (2, 2) window
        x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))
        # If the size is a square, you can specify with a single number
        x = F.max_pool2d(F.relu(self.conv2(x)), 2)
        x = self.dropout1(x)
        
        
       # x = F.max_pool2d(F.relu(self.conv3(x)), 2)
        x = torch.flatten(x, 1) # flatten all dimensions except the batch dimension
        x = F.relu(self.fc1(x))
        
        x = F.relu(self.fc2(x))
        x = self.dropout2(x)
        x =self.fc3(x)
        #print(x,torch.max(x))
        return x


net = Net()
net = net.float()
print(net)

This model returns a tensor of shape (N,2) with float type values in the range of [-1,1]. So, I took the index of the element with maximum value using torch.max(outputs,1).indices and my labels were negative and positive. So, I used the encoding : negative->0 and positive->1 .So,finally what goes to my loss=criterion() is : the output that is the index with maximum value and the label with the encoding as specified above.I wanted to confirm if this is right or not.Since,this is the first time I am building a model and doing binary classification.I also tried not to use torch.tensor(loss,requires-grad=True) but commenting this line of code brought up errors. I am actually not able to figure out why my weights are not getting updated and why the model is not getting trained. I do have a doubt that how to mark a label as 0 and the other as 1.Specially when ,they are compared to the indices of an array. Like instead of labelling positive as 0 if I mark it 1.Will that cause any problem?

$\endgroup$
0
$\begingroup$

You are calculating the loss, then clearing all the gradients, then optimising the function. The optimiser won't be able to update the weights if it has no gradients. Move the optimizer.zero_grad() either before calculating the loss, or after making an optimisation step. There's no need for converting the loss to a tensor requiring gradients because that's the default.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.