Softmax activation function is used to convert any random vector into a probability distribution. So, it is generally used as an activation function in the last layer of deep neural networks that are intended for classification.
softmax() does not satisfy the property of scale invariance i.e., the ratio of inputs and the ratio of outputs does not remain the same.
For example if we give the input
[1.4285, 0.3815] to
softmax, the function will give
[0.7402, 0.2598] as output.We can calculate ratios
1.4285: 0.3815 and
0.7402: 0.2598 and find that they are not the same.
Are there any scale-invariant versions of softmax?