0
$\begingroup$

I am new to machine learning and recently I joined a course where I was given a logistic regression assignment in which I had to split 20% of the training dataset for the validation dataset and then use the validation dataset to capture the minimum possible loss and then use the test dataset to find the accuracy of the model. Below is my code for implementing logistic regression

class LogReg(LinReg):
    def __init__(self, n_dim, bias=True):
        if bias:
            n_dim = n_dim + 1
        super(LogReg, self).__init__(n_dim)
        self.bias = bias
    
  def __call__(self, x):
      return x.mm(self.theta).sigmoid()

  def compute_loss(self, x, y, lambda_reg):
      # The function has a generic implementation, and can also work for the neural nets!
      predictions = self(x)
      loss = -(y * torch.log(predictions) + (1-y) * torch.log(1 - predictions)).mean()
      regularizer = self.theta.transpose(0, 1).mm(self.theta)
      return loss + regularizer.mul(lambda_reg)
  @staticmethod
  def add_bias(x):
      ones = torch.ones((x.size(0), 1), dtype=torch.float32)
      x_hat = torch.cat((ones, x), dim=-1)
      return x_hat

  def fit(self, x, y, num_iter=10, mb_size=32, lr=1e-1, lambda_reg=1e-2, reset=True):
      N = x.size(0)
      losses = []
      x_hat = x
      # Adding a bias term if needed
      if self.bias:
          x_hat = self.add_bias(x)
      if reset:
          self.reset() # Very important if you want to call fit multiple times
      num_batches = x.size(0) // mb_size
      # The outer loop goes over `epochs`
      # The inner loop goes over the whole training data
      for it in range(num_iter):
          loss_per_epoch = 0
          for batch_it in range(num_batches):
              # has been implemented for the linear model
              self.zero_grad()

              ind = torch.randint(0, N, (mb_size, 1)).squeeze()
              x_mb, y_mb = x_hat[ind, :], y[ind, :]

              loss = self.compute_loss(x_mb, y_mb, lambda_reg)

              loss.backward()
              self.theta.data = self.theta.data - lr*self.grad().data
              loss_per_epoch += loss.item()
          
          loss_per_epoch /= num_batches
          losses.append(loss_per_epoch)
      
      return losses

How should I use the validation set at the level of epoch to find the best loss?

$\endgroup$

1 Answer 1

1
$\begingroup$

So generally, when you seperate your training data to 80%-20% then you fit method should get 2 x,y. better to call them x_train,y_train, x_val, y_val or something similar.

Now its important you do the split before entering the fit, and not do it for each epoch or something alike.

Once you do that and the fit method should be something like:

def fit(self, x_train, y_train, x_val, y_val, num_iter=10, mb_size=32, lr=1e-1, lambda_reg=1e-2, reset=True):

Then you should, at the end of each epoch, test the performance of the model on the validation set entirely and calculate the desired metric for evaluation. If you improved it's better to save the current model. This is done repeatedly for each epoch until the end of the training and you will guarantee to have the model who gave you the best results on the validation set rather than on the training set, which might be overfitting it.

I will do it in a separate method with the following flow:

  1. iterate on each sample in the validation set
  2. for each one calculate the loss/metric
  3. append it to some list
  4. return the mean of that list

And if the average result is better from some previously saved one, save the new model

$\endgroup$
5
  • $\begingroup$ Thank you for your answer. What is the purpose of writing x_val and y_val inside fit? In the separate method can I pass the x_val and y_val and then compute the loss? $\endgroup$
    – DeepakS
    Oct 1, 2020 at 7:26
  • $\begingroup$ @DeepakSonker you need to do the separation beforehand, and at the moment you have x,y data but there is no distinction between the training and validation $\endgroup$
    – David
    Oct 1, 2020 at 7:30
  • $\begingroup$ How to use the best loss to test the test_data? how will I implement that loss in my code? I am really confused. I have not done this previously. $\endgroup$
    – DeepakS
    Oct 1, 2020 at 7:39
  • $\begingroup$ @DeepakSonker So you will save the current best loss/acc and for each time you evaluate the results on the validation set you will examine the new results. if its better save the model if not skip; or something similar $\endgroup$
    – David
    Oct 1, 2020 at 8:04
  • $\begingroup$ Can you provide me the reading material where I can find the implementation of this? $\endgroup$
    – DeepakS
    Oct 1, 2020 at 8:22

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .