0
$\begingroup$

Softmax activation function is used to convert any random vector into a probability distribution. So, it is generally used as an activation function in the last layer of deep neural networks that are intended for classification.

But, the softmax() does not satisfy the property of scale invariance i.e., the ratio of inputs and the ratio of outputs does not remain the same.

For example if we give the input [1.4285, 0.3815] to softmax, the function will give [0.7402, 0.2598] as output.We can calculate ratios 1.4285: 0.3815 and 0.7402: 0.2598 and find that they are not the same.

Are there any scale-invariant versions of softmax?

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ Could you explian why you need a scale-invariant function? Unmodified logits in a neural network can be, and often are negative. They cannot be simply scaled into a discrete probability distribution. However, if you are willing to accept other non-linearities to adjust this, then something may be possible. But it not clear whether what has driven your question, thus whether such a thing would be a good idea. $\endgroup$
    – Neil Slater
    Feb 16, 2022 at 7:07
  • $\begingroup$ @NeilSlater The text book I am reading mentioned about this property explicitly. So, I think there might be some significance and hence asked it. $\endgroup$
    – hanugm
    Feb 16, 2022 at 23:55
  • $\begingroup$ Can you give a reference to where it is mentiioned and in which content? As far as I know there is no function that could turn arbitrary logits into a discrete probability distribution without impacting their ratios. Non-linearity is required to deal with negative inputs. $\endgroup$
    – Neil Slater
    Feb 17, 2022 at 7:07

1 Answer 1

Reset to default
1
$\begingroup$

Can't you just divide the numbers by their sum? If your numbers could be negative, you can clamp them to a probability of zero by passing them through a RELU activation.

Improve this answer
$\endgroup$
2
  • $\begingroup$ I am asking for reputed ones in application. $\endgroup$
    – hanugm
    Feb 16, 2022 at 11:28
  • 1
    $\begingroup$ I don't think there is any other function which preserves the ratios of numbers [0, a, b] and scales them to sum to 1. You explicitly ask if there is such function, and I answered that yes there is and provided an example. $\endgroup$
    – NikoNyrh
    Feb 17, 2022 at 14:41

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .