I just stumbled upon the concept of neuron coverage, which is the ratio of activated neurons and total neurons in a neural network. But what does it mean for a neuron to be "activated"? I know what activation functions are, but what does being activated mean e.g. in the case of a ReLU or a sigmoid function?
$\begingroup$
$\endgroup$
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
0
A neuron is said activated when its output is more than a threshold, generally 0.
For examples : \begin{equation} y = Relu(a) > 0 \end{equation} when \begin{equation} a = w^Tx+b > 0 \end{equation}
Same goes for sigmoid or other activation functions.
$\begingroup$
$\endgroup$
The term "activated" is mostly used when talking about activation functions which only outputs a value (except 0) when the input to the activation function is greater than a certain treshold.
Especially when discussing ReLU the term "activated" may be used. ReLU will be "activated" when it's output is greater than 0, which is also when it's input is greater than 0.
Other activation functions, like sigmoid, always returns a value greater than 0, and doesn't have any special treshold. Therefore, the term "activated" is of less meaning here.
Even though we know little about them, the neurons in the brain also seems to have something which resembles an activation function with some kind of "activation treshold".
