1
$\begingroup$

Isn't it true that using max over a softmax will be much slower because there is not a smooth gradient?

Max basically zeros out the gradients of all the non-maximum values. Especially at the beginning of training, this means it is zeroing out potentially useful features simply because of random weight initialization. Wouldn't this drastically slow down the training in the beginning?

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ Could you give more details of the substitution that you are considering? There's a few different ways you could use max in place of softmax. Some won't work at all, let alone slowly. But there might be some limited ways in which you could use max and still be able to train a classifier $\endgroup$
    – Neil Slater
    Jul 9 '20 at 20:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.