# Why do we throw out negative ReLU value?

when using Rectified Linear Unit after convolution layers we have to have twice as much filters to be able to detect features (eg both left and right edge detector). Why do we just throw out negative output of the unit? As I understand ReLU should have two outputs - positive and negative

A straight ReLU activation function has only one output. It has two distinct sections of its input domain.

$$f(x)={ \begin{cases}0 & {\text{for }}x<0 \\ x & {\text{for }}x\geq 0\end{cases}}$$

Negative inputs produce zero output, so in that section of the domain, variance is not passed forward because the derivative of output with respect to input is zero. Positive inputs pass through, so in that section of the domain, variance is preserved because the derivative of output with respect to input is one. The derivative of output with respect to input for zero input is undefined, but addressing that condition is a separate topic.

There cannot be left or right edge detectors, or top and bottom ones either. Edge detection layers normally do not have access to inner or outer shape or object indications. Determining the likelihoods that a light gradient represents an edge of some contiguous object happens further down the pipeline in vision systems. Consider the mathematics. One can have a negative or positive partial derivative of lightness with respect to horizontal position, but the layer that would detect such cannot correlate positive or negative derivatives to left or right edges of any object.

That a ReLU activation function does not pass forward input variance because its input is negative is by design. That particular property can be leveraged to benefit convergence. The ReLU function is appropriate for some applications and not others because of that property or any one of several other properties. There are at least twenty other activation functions in use, and it is expected that more will be added.

Why [was] ReLU designed this way. Is there reason why it has only one output? Is there activation function that produces [the following]?

$$(0, -x) \text{ for } x < 0; (x, 0) \text{ for } x >= 0$$

That flexibility already exists using any two cells in a layer using ReLU or any other activation function, one with a positive attenuation factor in the parameter matrix leading into cell $$i$$ and another with a negative attenuation factor in the parameter matrix leading into cell $$j$$, where $$i \ne j$$. The artificial network design could be modified to have complementary functions in a single cell, but there is no particular advantage to offset the greater attenuation flexibility of the current layer design.

• So my question why ReLU was designed this way. Is there reason why it has only one output? Is there activation function that produces (0, -x) for x<0; (x, 0) for x>=0? – Andrew Matuk Jan 7 '19 at 10:00

it seems I have found, it has the name Concatenated ReLU (CReLU)

Concatenated ReLU has two outputs, one ReLU and one negative ReLU, concatenated together. In other words, for positive x it produces [x, 0], and for negative x it produces [0, x]. Because it has two outputs, CReLU doubles the output dimension.

there is also Negative CReLU, it seems difference is only the sign

NCReLU(x) = ( ρ(x) , −ρ(−x) )