Softmax activation function is used to convert any random vector into a probability distribution. So, it is generally used as an activation function in the last layer of deep neural networks that are intended for classification.
But, the softmax()
does not satisfy the property of scale invariance i.e., the ratio of inputs and the ratio of outputs does not remain the same.
For example if we give the input [1.4285, 0.3815]
to softmax
, the function will give [0.7402, 0.2598]
as output.We can calculate ratios 1.4285: 0.3815
and 0.7402: 0.2598
and find that they are not the same.
Are there any scale-invariant versions of softmax?