# Is there some kind of "weighted maximum" that allows the gradients to backpropagate? [closed]

I was wanting to add a maximum in my neural network, but this seems a bad thing to do since it kills the gradients to all but one of the inputs.

Is there some kind of "weighted maximum" that allows the gradients to backpropagate?

Edit: I had a two dimensional tensor (correlation matrix) I wanted to reduce to one dimension.

• "wanting to add a maximum in my neural network", do you mean as an activation function? But how? It's not clear where you would use this "max" function. Edit your post to clarify that.
– nbro
May 13 at 8:59

• @StefanPerko SoftPlus(x) = log(1 + exp(x)) is a smooth approximation to relu which unlike relu would allow the gradients to propagate backwards for x<0, but I doubt there would be any benefit since gradient decent doesn't get stuck with relu anyway, which is interesting 2 days ago