I'm new to machine learning (so excuse my nomenclature), and not being a python developer, I decided to jump in at the deep (no pun intended) end writing my own framework in C++.

In my current design I have given each neuron/cell the possiblity to have a different activation function. Is this a plausible design for a neural network? A lot of the examples I see use the same activation function for all neurons in a given layer.

Is there a model which may require this, or should all neurons in a layer use the same activation function? Would I be correct in using different activation functions for different layers in the same model, or would all layers have the same activation function within a model?


From here:

Using other activation functions don’t provide significant improvement in performance and tweaking them doesn’t provide any big improvement. So as per simplicity we use same activation function for most of the case in Deep Neural Networks.

  • $\begingroup$ Many thanks, so should the same activation function be used throughout the entire network/model, or is there still merit in using different activation functions for different layers in the same model? $\endgroup$ – lfgtm Dec 30 '19 at 0:37
  • $\begingroup$ @lfgtm Normally, there is an improvement in mixing activation functions across a network (keeping layers to just 1 though). An example of this, is in a CNN, where the net is all ReLU activations until the last layer where, for classification, it is a softmax activation. $\endgroup$ – Recessive Dec 30 '19 at 10:29
  • $\begingroup$ Many thanks this answers my questions. Kind Regards. $\endgroup$ – lfgtm Dec 30 '19 at 12:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.