1
$\begingroup$

I was wanting to add a maximum in my neural network, but this seems a bad thing to do since it kills the gradients to all but one of the inputs.

Is there some kind of "weighted maximum" that allows the gradients to backpropagate?

Edit: I had a two dimensional tensor (correlation matrix) I wanted to reduce to one dimension.

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ "wanting to add a maximum in my neural network", do you mean as an activation function? But how? It's not clear where you would use this "max" function. Edit your post to clarify that. $\endgroup$
    – nbro
    May 13 at 8:59

1 Answer 1

Reset to default
1
$\begingroup$

The maximum function is not smooth, since it's first derivative is not continuous.

Having non-smooth functions is generally a bad thing for neural networks, since they don't work nicely with gradient decent.

So what you want is a smooth approximation to these functions.

Logsumexp is the smooth approximation to the maximum function and so it is what you should use in a neural network, just like softmax is a smooth approximation to the argmax https://en.wikipedia.org/wiki/LogSumExp

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ People have been using ReLu functions just fine. Differentiable almost everywhere is probably fine for most applications. $\endgroup$
    – Stefan Perko
    2 days ago
  • $\begingroup$ @StefanPerko SoftPlus(x) = log(1 + exp(x)) is a smooth approximation to relu which unlike relu would allow the gradients to propagate backwards for x<0, but I doubt there would be any benefit since gradient decent doesn't get stuck with relu anyway, which is interesting $\endgroup$
    – Tom Huntington
    2 days ago

Not the answer you're looking for? Browse other questions tagged .