In deep learning, is it possible to use discontinuous activation functions (e.g. one with jump discontinuity)?
(My guess : for example, ReLU is non-differentiable at a single point, but it still has well-defined derivative. If an activation function has a jump discontinuity, then its derivative is supposed to have a delta function at that point. However, backpropagation process is incapable of considering that delta function, so the convex optimization process will have some problem?)