I have enrolled in a course that uses only one hidden layer, and that is the only layer that has activation functions. The model can be visualized as follows:
and here is a PyTorch implementation:
class MnistModel(nn.Module): """Feedfoward neural network with 1 hidden layer""" def __init__(self, in_size, hidden_size, out_size): super().__init__() # hidden layer self.linear1 = nn.Linear(in_size, hidden_size) # output layer self.linear2 = nn.Linear(hidden_size, out_size) def forward(self, xb): # Flatten the image tensors xb = xb.view(xb.size(0), -1) # Get intermediate outputs using hidden layer out = self.linear1(xb) # Apply activation function out = F.relu(out) # Get predictions using output layer out = self.linear2(out) return out
shouldn't the output layer also have activation functions?