3
$\begingroup$

Why does every neuron in a hidden layer of a multi-layer perceptron (MLP) typically have the same activation function as every other neuron in the same or other hidden layers (so I exclude the output layer, which typically has a different activation function) of the MLP? Is this a requirement, are there any advantages, or maybe is it just a rule of thumb?

Improve this question
$\endgroup$
0

1 Answer 1

Reset to default
2
$\begingroup$

As you stated, it's popular to have some form of a rectified linear unit (ReLU) activation in hidden layers and the output layer is often a softmax or sigmoid (depending also on the problem: multi-class or binary classification, respectively), which provides an output that can be viewed as a probability distribution.

You could generalize this further to blocks of different activation functions within the same layer. This is something I've thought about, haven't done, but imagine has been attempted. In some sense, the idea here would be to allow for a subsection of the network to develop a representation that may not be feasible otherwise. These different representations within the same layer would then be unified by subsequent layers as we move closer to the output.

Improve this answer
$\endgroup$
5
  • $\begingroup$ regarding your last paragraph that's what I'm thinking about. Is there ever a benefit to have a layer where each Perceptron has a different activation function? $\endgroup$
    – user8714896
    Jul 23, 2020 at 21:14
  • $\begingroup$ I updated the answer, maybe it addresses the question a bit better. I think the larger point is that anything is possible; neural networks are highly flexible models afterall. But the motivation for doing so is crucial – what would you hope to gain by having a block of neurons with alternate activations in the same layer? Why is this gain not possible without it? I suspect that answers that motivate such an architecture are hard to find. Further, as far as I'm aware, these aren't features that are readily available in existing software libraries. $\endgroup$
    – Greenstick
    Jul 23, 2020 at 21:16
  • $\begingroup$ @Greenstick I suspect with some problems where characteristics of the objective function are not obvious that experimenting with a mixture of activation functions might afford a better approximation to the function trying to be learned. $\endgroup$
    – user8714896
    Jul 23, 2020 at 23:10
  • $\begingroup$ @user8714896 This is in effect done. Certain activations (e.g. PReLU) take in parameters that have their own associated weights (multiple params could exist for a layer); the neural net then learns how to parameterize these activations to minimize the objective function. You lose some interpretability and (maybe) motivation doing this (e.g. the biological motivation discussed above), but the nice part is that you don't have to precisely know the activation ahead of time – the neural net figures it out. Hope that helps; if this is in line with the answer you're looking for, please accept it. $\endgroup$
    – Greenstick
    Jul 29, 2020 at 20:07
  • $\begingroup$ This answer would improve considerably if you could do a little bit of research to find some research papers that show that, indeed, different activation functions can be used for different hidden neurons. $\endgroup$
    – nbro
    Jan 17, 2021 at 16:26

Not the answer you're looking for? Browse other questions tagged .