I am trying to implement Contractive auto-encoders in PyTorch but I don't know what I'm doing is right or not. The architecture of the auto-encoder is given below:

class AE(nn.Module):
    def __init__(self):
        super(AE, self).__init__()
        self.encoder = nn.Sequential(nn.Linear(784, 256),nn.Linear(256, 128), nn.Linear(128, 64))
        self.decoder = nn.Sequential(nn.Linear(64, 128), nn.Linear(128, 256), nn.Linear(256, 784))
        self.sigmoid = nn.Sigmoid()

    def forward(self, input):
        h1 = self.encoder(input)
        h2 = self.decoder(h1)
        sigmoid = self.sigmoid(h2)
       return h1,sigmoid

I am trying to implement the contractive loss function the code for which is given below:

mse_loss = nn.MSELoss()
lam= 1e-3

def contractive_loss(W, x, recons_x, h, lam):
    mse = mse_loss(recons_x, x)
    dh = h*(1-h)
    w_sum = torch.sum(W**2, dim=1)
    w_sum = w_sum.unsqueeze(1)
    contractive_loss_value = torch.sum(torch.mm(dh**2, w_sum),0)
    return mse + contractive_loss_value.mul(lam)

The training module is given below:

def train(model=ae_model, epoch=0, train_loader= train_loader):
    train_loss= 0
    total= 0

    for i, (data, label) in enumerate(train_loader):
        data= data.to(device).view(-1, 28*28)
        label = label.to(device).view(-1, 28*28)
        hidden_representation, recons_x = model(data)

        W = model.state_dict()['decoder.2.weight']
        loss = contractive_loss(W , label, recons_x, hidden_representation, lam)

        train_loss += loss.item()
        total += label.size(1)

Any help will be appreciated.

  • Using many linear layers without af is equivalent of using the one linear layer. I guess you r making AE for some MNIST images, so i ll post my (?? is questional and must be tested to do it or not): encoder : nn.Linear(784, 784/2), batch norm1d , torch.nn.Tanh, nn.Linear(784/2, 200),batch norm, torch.nn.Tanh, nn.Linear(200, 40) 30, 20 ?? batch norm ??, torch.nn.Tanh, try some dropout in places you want.
  • 64 is too thick for encoder, you do try to minimize the layer
  • Ahh, also ls 1e-3 may be too high but it must be tested.

  • decoder: nn.Linear(40, 200), batch norm1d, torch.nn.Tanh, nn.Linear( 200, 784/2,), batch norm1d ?? , nn.Linear(784/2, 784), torch.nn.sigmoid

  • try with MSE and BSE losses what coes better but MSE is actually fine

  • data= data.to(device).view(-1, 28*28) label = label.to(device).view(-1, 28*28) Dont copy the data to device in a train loop, make it once

  • contractive_loss: what kind of loos it that ? can you give me a link pls i d try it ? AFAIK its not so important, even MSE works fine.

  • Full connected setup is generaly bad for images. Try to move image 1-2 pixels, it will destroy the run. Use convo instead, something like i have :

    class ConvAE(torch.nn.Module):

    def __init__(self, in_channels):
        super(ConvAE, self).__init__()
    self.in_channels = in_channels
    self.af = torch.nn.ReLU()
    self.code_layer = None
    self.channels1 = 8
    self.channels2 = 6
    self.channels3 = 4
     def update_seq(self):
    self.seq_encode = torch.nn.Sequential(
        torch.nn.Conv2d(in_channels = self.in_channels, out_channels = self.channels1, kernel_size = 4, padding = 0),            
        torch.nn.BatchNorm2d(num_features =self.channels1),
        torch.nn.Conv2d(in_channels = self.channels1, out_channels = self.channels1, kernel_size = 4, padding = 0 ), 
        torch.nn.BatchNorm2d(num_features =self.channels1),
        torch.nn.Conv2d(in_channels = self.channels1, out_channels = self.channels2, kernel_size = 4, padding = 0 ),            
        torch.nn.BatchNorm2d(num_features =self.channels2),
        torch.nn.Conv2d(in_channels = self.channels2, out_channels = self.channels2, kernel_size = 4, padding = 0 ), 
        torch.nn.BatchNorm2d(num_features =self.channels2),
        torch.nn.Conv2d(in_channels = self.channels2, out_channels = self.channels3, kernel_size = 4, padding = 0 ),
        torch.nn.BatchNorm2d(num_features =self.channels3),
    self.seq_decode = torch.nn.Sequential(
        torch.nn.ConvTranspose2d( self.channels3 ,self.channels2,4),  
        torch.nn.ConvTranspose2d( self.channels2 ,self.channels2,4),
        torch.nn.ConvTranspose2d( self.channels2 ,self.channels1,4),
        torch.nn.ConvTranspose2d( self.channels1 ,self.channels1,4),
        torch.nn.ConvTranspose2d( self.channels1 , self.in_channels,4),
     def forward_encode(self, x):
         self.code_layer =  self.seq_encode(x)
         return self.code_layer
         def forward_decode(self):
             return self.seq_decode(self.code_layer)
         def forward(self, x):
              return self.forward_decode()

Weight should be W = model.state_dict()['encoder.2.weight']

  • 2
    $\begingroup$ Hi @Vaan welcome to AI StackExchange! I think your answer is too short, please give additional information something like the reason why you think this is the answer $\endgroup$ – malioboro Jul 15 at 9:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.