Why does every neuron in a multi-layer perceptron typically have the same activation function? Is this a requirement, are there any advantages, or maybe is it just a rule of thumb?


They don't. In fact, if we look to similar models like convolutional neural nets, we see that it's popular to have some form of a rectified linear unit (ReLU) activation in earlier layers and the output layer is often a softmax activation, which provides an output that can be viewed as a probability distribution.

Generally, it depends on what you're trying to achieve. For a long time, the logistic (sigmoid) activation was popular because it was thought to be similar to the synaptic potentials seen in the neurons of the human brain. But it turns out that the this activation function has some issues, such as vanishing gradients. One way to get around the vanishing gradient problem is to use a different activation – in effect discarding the biological motivation of the model for greater stability during training.

You could generalize this further to blocks of different activation functions within the same layer. This is something I've thought about, haven't done, but imagine has been attempted. In some sense, the idea here would be to allow for a subsection of the network to develop a representation that may not be feasible otherwise. These different representations within the same layer would then be unified by subsequent layers as we move closer to the output.

| improve this answer | |
  • 4
    $\begingroup$ I don't think the OP was referring to the output layer. I think he was trying to understand why all hidden neurons typically have the same activation function (but I may be wrong too). $\endgroup$ – nbro Jul 23 at 21:11
  • $\begingroup$ @nbro yes what nbro said. $\endgroup$ – user8714896 Jul 23 at 21:12
  • $\begingroup$ regarding your last paragraph that's what I'm thinking about. Is there ever a benefit to have a layer where each Perceptron has a different activation function? $\endgroup$ – user8714896 Jul 23 at 21:14
  • $\begingroup$ I updated the answer, maybe it addresses the question a bit better. I think the larger point is that anything is possible; neural networks are highly flexible models afterall. But the motivation for doing so is crucial – what would you hope to gain by having a block of neurons with alternate activations in the same layer? Why is this gain not possible without it? I suspect that answers that motivate such an architecture are hard to find. Further, as far as I'm aware, these aren't features that are readily available in existing software libraries. $\endgroup$ – Greenstick Jul 23 at 21:16
  • $\begingroup$ @Greenstick I suspect with some problems where characteristics of the objective function are not obvious that experimenting with a mixture of activation functions might afford a better approximation to the function trying to be learned. $\endgroup$ – user8714896 Jul 23 at 23:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.