1
$\begingroup$

An activation function is a function from $R \rightarrow R$. It takes as input the inner products of weights and activations in the previous layer. It outputs the activation.

A softmax however, is a function that takes input from $R^p$, where $p$ is the number of possible outcomes that need to be classified. Therefore, strictly speaking, it cannot be an activation function.

Yet everywhere on the net it says the softmax is an activation function. Am I wrong or are they?

|improve this question
$\endgroup$
  • 1
    $\begingroup$ An activation function isn't necessarily defined as $\mathbb{R} \rightarrow \mathbb{R}$. $\endgroup$ – nbro Feb 13 at 11:56
  • 1
    $\begingroup$ I would be interested to know where you are taking your definition of an activation function from in your first paragraph? I am not sure if it has a formal definition, and even if it did, this would maybe become an issue of mis-naming or mis-classifying something, but the engineering of it - what it is and does - would remain exactly the same. $\endgroup$ – Neil Slater Feb 13 at 14:27
  • $\begingroup$ Coming to think of it, I did not see a formal definition. Instead, I saw pictures such as those link, where it seems that they are functions that take from & $\mathbb{R}$. $\endgroup$ – TimvanSch Feb 22 at 11:09
1
$\begingroup$

I see no problem in regarding the softmax as a particular activation function which takes a vector input and produces a vector output. In fact, the sigmoid function can be viewed as a two-dimensional softmax in which one of the two inputs is hardwired to zero while the corresponding output is neglected.

|improve this answer
$\endgroup$
  • $\begingroup$ Thanks for your answer. Indeed, there is no problem with regarding it as such. The question was mostly aimed at clarification. $\endgroup$ – TimvanSch Feb 22 at 11:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.