2
$\begingroup$

I have enrolled in a course that uses only one hidden layer, and that is the only layer that has activation functions. The model can be visualized as follows:

enter image description here

and here is a PyTorch implementation:

class MnistModel(nn.Module):
    """Feedfoward neural network with 1 hidden layer"""
    def __init__(self, in_size, hidden_size, out_size):
        super().__init__()
        # hidden layer
        self.linear1 = nn.Linear(in_size, hidden_size)
        # output layer
        self.linear2 = nn.Linear(hidden_size, out_size)
        
    def forward(self, xb):
        # Flatten the image tensors
        xb = xb.view(xb.size(0), -1)
        # Get intermediate outputs using hidden layer
        out = self.linear1(xb)
        # Apply activation function
        out = F.relu(out)
        # Get predictions using output layer
        out = self.linear2(out)
        return out

shouldn't the output layer also have activation functions?

$\endgroup$
5
  • $\begingroup$ All layers (except input) have an activation. If it is not defined explicitly, it will be the linear activation. Depending on your task, you may need the linear activation (e.g., for regression) $\endgroup$ Aug 1 at 14:55
  • $\begingroup$ @ArayKarjauv Thank you for replying. In the course I'm doing, there is no activation function on the output layer. Do you know why is that? $\endgroup$
    – Omar Zayed
    Aug 1 at 18:29
  • $\begingroup$ can you please provide a piece of additional information or an example? $\endgroup$ Aug 1 at 19:25
  • $\begingroup$ Well, this isn't strictly true. For example, I have some regression networks where the output layer is designed to not have an activation. It really depends on what you're doing. $\endgroup$ Aug 1 at 20:49
  • $\begingroup$ @ArayKarjauv jovian.ai/kukuquack/04-feedforward-nn this is my notebook, in one of the code cells, there are 2 linear layers and one activation function for the first layer, You will find it in the mnistmodel class, forward function. $\endgroup$
    – Omar Zayed
    Aug 2 at 9:45
2
$\begingroup$

A neural network layer with no activation function is the same as "linear activation" i.e. $f(x) = x$

This is often used for the output layer in regression problems, where a constrained output like that of sigmoid, hyperbolic tangent or ReLU may not be appropriate.

For an output layer, this is fine, and does not conflict with any theory behind neural networks. It will often be combined with a mean squared error (MSE) loss function that makes the gradient calculation simple at the output layer.

For hidden layers, skipping the activation function can be a problem, since a purely linear layer in the middle of a multi-layer network is redundant - it could be replaced or even removed with no impact. That is because two directly connected linear layers are functionally equivalent to a single linear layer with different parameters, and every hidden layer consists of a linear component plus an activation function. So even one missing activation function on a hidden layer directly connects two linear sub-components, making one of them redundant.

In the case of a classifier, some libraries (notably including PyTorch which you are using) require you to have two variants of your neural network - a trainable version without a final sigmoid or softmax layer, and the "full" version which adds sigmoid or softmax on top (usually just a function that calls the training network and add this last activation). This is done to allow you to use more stable gradient calculations that are based on the logits, the values before applying an activation function.

$\endgroup$
5
  • $\begingroup$ Thanks for this. Exactly the point I was making in my comment. $\endgroup$ Aug 2 at 13:11
  • $\begingroup$ While this is correct, it does not answer the question. The problem considered in the example is multiclass classification with cross-entropy loss. As far as I know, it requires softmax activation. Will it work with linear activation? $\endgroup$ Aug 2 at 13:35
  • 1
    $\begingroup$ @ArayKarjauv Actually yes it will, sort of. It is common in PyTorch to not implement the final softmax, and to use a more stable loss function gradient function on the logits. Then there is also an extra method added to the nn module that runs the softmax when needed. So you will often see an incomplete variant of your network used for training purposes only. I added a para about that $\endgroup$ Aug 2 at 13:51
  • $\begingroup$ I see. So the network does have softmax activation for the output layer, which is integrated into F.cross_entropy $\endgroup$ Aug 2 at 14:08
  • $\begingroup$ @ArayKarjauv That's not the specific function I recall using, I would use pytorch.org/docs/stable/generated/… - but I think they are very similar just with different calling conventions, and likely resolve to the same optimised C code underneath $\endgroup$ Aug 2 at 14:47
2
$\begingroup$

I'd like to add more information to Neal's answer.

A network implemented in the example does not include activation for the output, because it will be applied during the training:

# ...
def training_step(self, batch):
    images, labels = batch 
    out = self(images)
    # apply activation and calculate loss
    loss = F.cross_entropy(out, labels)
    return loss
# ...

The problem considered in the example is multiclass classification, which is usually solved with softmax activation and cross-entropy loss. F.cross_entropy combines log_softmax and nll_loss in a single function, which is numerically more stable as softmax and NLL loss. See this discussion for more details, and here is an article about different implementations.

$\endgroup$
6
  • $\begingroup$ But why isn't there the activation function on the output layer during calling the forward function? @ArayKarjauv $\endgroup$
    – Omar Zayed
    Aug 2 at 16:47
  • $\begingroup$ @OmarZayed That is unnecessary. PyTorch allows additional calculations (e.g., mean, normalization, multiplication, etc.) to be done outside of the model class. $\endgroup$ Aug 2 at 17:04
  • $\begingroup$ So when is adding an activation function necessary in the output layer? $\endgroup$
    – Omar Zayed
    Aug 3 at 12:43
  • $\begingroup$ @OmarZayed when you use the negative log-likelihood loss alone or calculate it manually (in that case you can also apply softmax manually outside the forward method). Please read this article for more information. $\endgroup$ Aug 3 at 14:05
  • $\begingroup$ so according to my understanding, the output layer shouldn't have an activation function because the only thing we care about is getting the highest value for some label? $\endgroup$
    – Omar Zayed
    Aug 3 at 20:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.