How to use validation dataset in my logistic regression model?

Question

I am new to machine learning and recently I joined a course where I was given a logistic regression assignment in which I had to split 20% of the training dataset for the validation dataset and then use the validation dataset to capture the minimum possible loss and then use the test dataset to find the accuracy of the model. Below is my code for implementing logistic regression

class LogReg(LinReg):
    def __init__(self, n_dim, bias=True):
        if bias:
            n_dim = n_dim + 1
        super(LogReg, self).__init__(n_dim)
        self.bias = bias
    
  def __call__(self, x):
      return x.mm(self.theta).sigmoid()

  def compute_loss(self, x, y, lambda_reg):
      # The function has a generic implementation, and can also work for the neural nets!
      predictions = self(x)
      loss = -(y * torch.log(predictions) + (1-y) * torch.log(1 - predictions)).mean()
      regularizer = self.theta.transpose(0, 1).mm(self.theta)
      return loss + regularizer.mul(lambda_reg)
  @staticmethod
  def add_bias(x):
      ones = torch.ones((x.size(0), 1), dtype=torch.float32)
      x_hat = torch.cat((ones, x), dim=-1)
      return x_hat

  def fit(self, x, y, num_iter=10, mb_size=32, lr=1e-1, lambda_reg=1e-2, reset=True):
      N = x.size(0)
      losses = []
      x_hat = x
      # Adding a bias term if needed
      if self.bias:
          x_hat = self.add_bias(x)
      if reset:
          self.reset() # Very important if you want to call fit multiple times
      num_batches = x.size(0) // mb_size
      # The outer loop goes over `epochs`
      # The inner loop goes over the whole training data
      for it in range(num_iter):
          loss_per_epoch = 0
          for batch_it in range(num_batches):
              # has been implemented for the linear model
              self.zero_grad()

              ind = torch.randint(0, N, (mb_size, 1)).squeeze()
              x_mb, y_mb = x_hat[ind, :], y[ind, :]

              loss = self.compute_loss(x_mb, y_mb, lambda_reg)

              loss.backward()
              self.theta.data = self.theta.data - lr*self.grad().data
              loss_per_epoch += loss.item()
          
          loss_per_epoch /= num_batches
          losses.append(loss_per_epoch)
      
      return losses

How should I use the validation set at the level of epoch to find the best loss?

Answer 1

So generally, when you seperate your training data to 80%-20% then you fit method should get 2 x,y. better to call them x_train,y_train, x_val, y_val or something similar.

Now its important you do the split before entering the fit, and not do it for each epoch or something alike.

Once you do that and the fit method should be something like:

def fit(self, x_train, y_train, x_val, y_val, num_iter=10, mb_size=32, lr=1e-1, lambda_reg=1e-2, reset=True):

Then you should, at the end of each epoch, test the performance of the model on the validation set entirely and calculate the desired metric for evaluation. If you improved it's better to save the current model. This is done repeatedly for each epoch until the end of the training and you will guarantee to have the model who gave you the best results on the validation set rather than on the training set, which might be overfitting it.

I will do it in a separate method with the following flow:

1. iterate on each sample in the validation set
2. for each one calculate the loss/metric
3. append it to some list
4. return the mean of that list

And if the average result is better from some previously saved one, save the new model