Suppose I want to predict the position of a sensor based on its reading.
I can first predict the unit vector and predict the distance to be multiplied to this vector. And I know that distance will never be negative because all the negative parts are inside unit vector already.
Should I apply ReLU to the distance before multiplying it to the unit vector?
I'm thinking that this can be helpful to eliminate the network from needing too much training data by restricting the output ranges the network could give. But I also think that it could make the learning slower when the ReLU unit dies (value=0) so the gradient doesn't flow properly somehow.