0
$\begingroup$

Below I attach an image of accuracy curves. I got a lot of suggestions regarding some improvement in below curves. Following are my experiments in order to make the curve stable-->

  1. I used lr = 4.44e-5
  2. Weight decay = 3e-6
  3. Used BatchNorm and Dropouts in architecture

I feel that weight updates are very sensitive to validation data, but not very sensitive to train data. I would appreciate any suggestions for making validation curve stable (currently just concerned with stability, not with overfitting or under-fitting)

My model is 1.3 M parameters large. Total dataset = 13,000 hyper-spectral images of wheat seads. Test_size = 20% Train size = 55% Validation size = 25%

Here is my model architecture :

class HSIModel(nn.Module):
  def __init__(self , config, n_res_blocks = 12):
    super(HSIModel, self).__init__()
    self.config = config
    self.in_channels = self.config['in_channels']
    self.n_res_blocks = n_res_blocks
    self.squeeze_channels = 512
    self.model = self.get_model()
    self.layer_lr = [{'params' : self.base_model.parameters()},{'params': self.head.parameters(), 'lr': self.config['lr'] * 1}]


  def get_model(self):
    self.head = nn.Sequential(
                  nn.Flatten(1) ,
                  Dense(0.4, 512, 128), 
                  # Dense(0.15, 256, 128), 
    )
    li = [ResidualBlock(256) for i in range(self.n_res_blocks)]
    self.base_model = nn.Sequential(
        # BandAttentionBlock(self.in_channels), 
        SqueezeBlock(self.in_channels, 100),
        SqueezeBlock(100, 512),
        nn.Dropout(p=0.2) ,
        XceptionBlock(512, 256), 
        XceptionBlock(256, 256), 
        nn.Dropout(p=0.2) ,
        # XceptionBlock(256, 128),
        nn.Sequential(*li), 
        # XceptionBlock(128, 256), 
        # SeparableConvBlock(128, 256), 

        SeparableConvBlock(256,512), 
        nn.MaxPool2d(kernel_size = (3,3) ,stride = (2,2)) , 
        nn.Dropout(p=0.15) ,
        nn.AdaptiveAvgPool2d((1,1)) ,
    )
    return nn.Sequential(
                  self.base_model,
                  self.head, 
                  Dense(0, 128, self.config['num_classes']),
                        )

  def forward(self, x):

    return self.model(x)

Below I also attach the code file where modules used in above architecture are written. Please also comment on architecture.

Pls help out, if you need any other information, pls mentionenter image description here

The modules used in above model have their usual architecture.

$\endgroup$
3
  • $\begingroup$ what about the loss $\endgroup$
    – Alberto
    Commented Dec 10, 2023 at 22:02
  • $\begingroup$ Similar behaviour $\endgroup$ Commented Dec 11, 2023 at 9:20
  • $\begingroup$ Hi @SarvagyaPorwal and welcome to AI Stack Exchange! If possible, please outline a specific question for this post. It may help you get answers more quickly. $\endgroup$
    – DeepQZero
    Commented Dec 19, 2023 at 20:22

0

You must log in to answer this question.