Should I apply ReLU to non negative output?

Question

Suppose I want to predict the position of a sensor based on its reading.

I can first predict the unit vector and predict the distance to be multiplied to this vector. And I know that distance will never be negative because all the negative parts are inside unit vector already.

Should I apply ReLU to the distance before multiplying it to the unit vector?

I'm thinking that this can be helpful to eliminate the network from needing too much training data by restricting the output ranges the network could give. But I also think that it could make the learning slower when the ReLU unit dies (value=0) so the gradient doesn't flow properly somehow.

Answer 1

Applying a ReLU activation function to a non-negative number is the same as applying the identity activation function.

What you may wish to do is apply $\ln(|\vec{x}|)$, $\arctan(x[0], x[1])$, ..., $\arctan(x[0], x[N-1])$, where $N$ is the number of dimensions to $\vec{x}$, and pass this directly into a parameter matrix multiplication (which involves deliberately using an identity activation in the input layer in some frameworks). This typically improves speed, accuracy, and reliability of convergence when the magnitudes exhibit a distribution close to an exponential of Gaussian and the directional components are near Gaussian when expressed in radians.