2
$\begingroup$

Consider the following simple neural network with only one neuron.

  • The input is $x_1$ and $y_2$, where $-250 < x < 250$ and $-250 < y < 250$
  • The weights of the only neuron are $w_1$ and $w_1$
  • The output of the neuron is given by $o = \sigma(x_1w_1 + x_2w_2 + b)$, where $\sigma$ is the ReLU activation function and $b$ the bias.
  • Thus the cost should be $(o - y)^2$.

When using the sigmoid activation function, the target for each point is usually $0$ or $1$.

But I'm a little confused witch target to use when the activation function is the ReLU, given that it can output numbers greater than 1.

$\endgroup$
  • 1
    $\begingroup$ Hi and welcome to AI SE! Why do you want to use the ReLU as the activation function of the last layer of your neural network? $\endgroup$ – nbro Mar 10 at 21:01
  • $\begingroup$ Thanks for editing! I'm a beginner with neural network. Actually, I read that ReLU is better than sigmoid because it have a non-linear behavior. Should I use sigmoid at the output layer? $\endgroup$ – Alan Vinícius Mar 11 at 1:05
  • 1
    $\begingroup$ ReLU and sigmoid have different properties, as you already noticed. I've never seen the ReLU being used as the activation function of the output layer (but I don't exclude people to use it for some reason, I just never saw it). A sigmoid is often used because the sigmoid squashes its inputs to the range $[0, 1]$, so, by doing this, the network will produce numbers that can be interpreted as probabilities, which can be useful in certain cases. In general, the targets don't necessarily need to be restricted to be 0 or 1. Do you want me to provide this information as a formal answer below? $\endgroup$ – nbro Mar 11 at 1:08
  • $\begingroup$ Got it! You were very clear. $\endgroup$ – Alan Vinícius Mar 11 at 1:31
1
$\begingroup$

ReLU and sigmoid have different properties (i.e. range), as you already noticed. I've never seen the ReLU being used as the activation function of the output layer (but some people may use it for some reason, e.g. regression tasks where the output needs to be positive). ReLU is usually used as the activation function of a hidden layer. However, in your case, you don't have hidden layers.

The sigmoid function is used as the activation function of the output layer when you need to interpret the output of the neural network as a probability, i.e. a number between $0$ and $1$, given that the sigmoid function does exactly this, i.e. it squashes its input to the range $[0, 1]$, i.e. $\text{sigmoid}(x) = p \in [0, 1]$. When do you need the output of the network to be a probability? For example, if you decide to use the cross-entropy loss function (which is equivalent to the negative log-likelihood), then the output of your network should be a probability. For example, if you need to solve a binary classification task, then the combination of a sigmoid as the activation function of the output layer and the binary cross-entropy as the loss function is probably what you need.

You could also have a classification problem with more than 2 classes (multi-class classification problem). In that case, you probably need to use a softmax as the activation function of your network combined with a cross-entropy loss function.

See this question How to choose cross-entropy loss in TensorFlow? on Stack Overflow for more info about different cross-entropy functions.

By the way, in general, the targets don't necessarily need to be restricted to be 0 or 1. For example, if you are solving a regression task, your target may just be any number. However, in that case, you may need another loss function (which is often the mean squared error).

| improve this answer | |
$\endgroup$
1
$\begingroup$

You are misunderstanding something. You are mixing up inner layers with the output layer. But the question was very good.

Fist of all, with the only one layer and one neuron neural networks it does not exist. Only one layer can not bring nonlinearity in the network. One neuron network means it's a linear regression or logistic regression if it passes through a sigmoid activation.

You have to look at NN like below. The output of a neural networks is the final layer output. There are two possible situations in the model.

  1. Classification model: For classification people generally use softmax and it has more than one output and the maximum value always below 1(It's a probability distribution).

  2. Regression model: A regression model has a continuous output like the output you mentioned in your problem statement. It has only one output channel. In the regression model, you can make the output as a linear combination of the previous layers (without defining the output range). You can use ReLU if you are sure your prediction is always positive.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.