How to determine the target value when using ReLU as activation function?

Question

Consider the following simple neural network with only one neuron.

• The input is $x_1$ and $y_2$, where $-250 < x < 250$ and $-250 < y < 250$
• The weights of the only neuron are $w_1$ and $w_1$
• The output of the neuron is given by $o = \sigma(x_1w_1 + x_2w_2 + b)$, where $\sigma$ is the ReLU activation function and $b$ the bias.
• Thus the cost should be $(o - y)^2$.

When using the sigmoid activation function, the target for each point is usually $0$ or $1$.

But I'm a little confused witch target to use when the activation function is the ReLU, given that it can output numbers greater than 1.

Answer 1

ReLU and sigmoid have different properties (i.e. range), as you already noticed. I've never seen the ReLU being used as the activation function of the output layer (but some people may use it for some reason, e.g. regression tasks where the output needs to be positive). ReLU is usually used as the activation function of a hidden layer. However, in your case, you don't have hidden layers.

The sigmoid function is used as the activation function of the output layer when you need to interpret the output of the neural network as a probability, i.e. a number between $0$ and $1$, given that the sigmoid function does exactly this, i.e. it squashes its input to the range $[0, 1]$, i.e. $\text{sigmoid}(x) = p \in [0, 1]$. When do you need the output of the network to be a probability? For example, if you decide to use the cross-entropy loss function (which is equivalent to the negative log-likelihood), then the output of your network should be a probability. For example, if you need to solve a binary classification task, then the combination of a sigmoid as the activation function of the output layer and the binary cross-entropy as the loss function is probably what you need.

You could also have a classification problem with more than 2 classes (multi-class classification problem). In that case, you probably need to use a softmax as the activation function of your network combined with a cross-entropy loss function.

See this question How to choose cross-entropy loss in TensorFlow? on Stack Overflow for more info about different cross-entropy functions.

By the way, in general, the targets don't necessarily need to be restricted to be 0 or 1. For example, if you are solving a regression task, your target may just be any number. However, in that case, you may need another loss function (which is often the mean squared error).

Answer 2

You are misunderstanding something. You are mixing up inner layers with the output layer. But the question was very good.

Fist of all, with the only one layer and one neuron neural networks it does not exist. Only one layer can not bring nonlinearity in the network. One neuron network means it's a linear regression or logistic regression if it passes through a sigmoid activation.

You have to look at NN like below. The output of a neural networks is the final layer output. There are two possible situations in the model.

1. Classification model: For classification people generally use softmax and it has more than one output and the maximum value always below 1(It's a probability distribution).

2. Regression model: A regression model has a continuous output like the output you mentioned in your problem statement. It has only one output channel. In the regression model, you can make the output as a linear combination of the previous layers (without defining the output range). You can use ReLU if you are sure your prediction is always positive.