From : https://debuggercafe.com/implementing-deep-autoencoder-in-pytorch/ the following autoencoder is defined
class Autoencoder(nn.Module): def __init__(self): super(Autoencoder, self).__init__() # encoder self.enc1 = nn.Linear(in_features=784, out_features=256) self.enc2 = nn.Linear(in_features=256, out_features=128) self.enc3 = nn.Linear(in_features=128, out_features=64) self.enc4 = nn.Linear(in_features=64, out_features=32) self.enc5 = nn.Linear(in_features=32, out_features=16) # decoder self.dec1 = nn.Linear(in_features=16, out_features=32) self.dec2 = nn.Linear(in_features=32, out_features=64) self.dec3 = nn.Linear(in_features=64, out_features=128) self.dec4 = nn.Linear(in_features=128, out_features=256) self.dec5 = nn.Linear(in_features=256, out_features=784) def forward(self, x): x = F.relu(self.enc1(x)) x = F.relu(self.enc2(x)) x = F.relu(self.enc3(x)) x = F.relu(self.enc4(x)) x = F.relu(self.enc5(x)) x = F.relu(self.dec1(x)) x = F.relu(self.dec2(x)) x = F.relu(self.dec3(x)) x = F.relu(self.dec4(x)) x = F.relu(self.dec5(x)) return x net = Autoencoder()
Autoencoder class, we can see that 784 features are passed through a series of transformations and are converted to 16 features.
The transformations (
out_features) for each layer are:
784 to 256 256 to 128 128 to 64 64 to 32 32 to 16
Why do we perform this sequence of operations? For example, why don't we perform the following sequence of operations instead?
784 to 256 256 to 128
784 to 512 512 to 256 256 to 128
Or maybe just encode in two layers:
784 to 16
Does the reduction of the dimensions over multiple layers (instead of a single layer) allow more details to be stored within the final representation? For example, if we used only the transformation $784 \rightarrow 16$, may this cause some detail not to be encoded? If so, why is this the case?