Isn't it true that using max over a softmax will be much slower because there is not a smooth gradient?

Max basically zeros out the gradients of all the non-maximum values. Especially at the beginning of training, this means it is zeroing out potentially useful features simply because of random weight initialization. Wouldn't this drastically slow down the training in the beginning?

  • 3
    $\begingroup$ Could you give more details of the substitution that you are considering? There's a few different ways you could use max in place of softmax. Some won't work at all, let alone slowly. But there might be some limited ways in which you could use max and still be able to train a classifier $\endgroup$ Commented Jul 9, 2020 at 20:19


You must log in to answer this question.