Questions tagged [activation-function]

For questions related to the selection of and theory behind specific activation functions used in artificial networks.

Filter by
Sorted by
Tagged with
23
votes
6answers
12k views

What is the purpose of an activation function in neural networks?

It is said that activation functions in neural networks help introduce non-linearity. What does this mean? What does non-linearity mean in this context? How does the introduction of this non-...
15
votes
3answers
7k views

How to choose an activation function?

I choose the activation function for the output layer depending on the output that I need and the properties of the activation function that I know. For example, I choose the sigmoid function when I'm ...
10
votes
1answer
1k views

How exactly can ReLUs approximate non-linear and curved functions?

Currently, the most commonly used activation functions are ReLUs. So I answered this question What is the purpose of an activation function in neural networks? and, while writing the answer, it struck ...
10
votes
5answers
3k views

Why do activation functions need to be differentiable in the context of neural networks?

Why should an activation function of a neural network be differentiable? Is it strictly necessary or is it just advantageous?
4
votes
1answer
408 views

What is a “logit probability”?

DeepMind's paper "Mastering the game of Go without human knowledge" states in its "Methods" section on its "Neural network architecture" that the output layer of AlphaGo Zero's policy head is "A fully ...
2
votes
0answers
127 views

Non-monotonic activation function and XOR problem with perceptron

I already post the question on Stackoverflow : https://stackoverflow.com/questions/53785922/is-the-use-of-a-non-monotonic-activation-function-is-a-correct-and-viable-solut?noredirect=1#...