1
$\begingroup$

I am building my first ANN from scratch. I know that I need a transfer function and I want to use the sigmoid function as my teacher recommended that. That function can be between 0 and 1, but my input values for the network are between -5 and 20. Someone told me that I need to scale the function so that it is in the range of -5 and 20 instead of 0 and 1. Is this true? Why?

$\endgroup$
2
  • $\begingroup$ I believe it's not true because the scaling is done by the weights, which are learned by the ANN during the optimization. Hence, it's the ANN job, not ours. $\endgroup$
    – Javier TG
    Nov 20 '20 at 9:21
  • $\begingroup$ The sigmoid function can take any value but it is recommended that inputs are scaled to be lower. The most common sigmoid function can only output values between 0 and 1. If you mean that the values supposed to be outputted by then network are between -5 and 20 then you could add 5 and divide the values by 25 and reverse that after training. Whoever said that the input values needed to be between -5 and 20 was wrong or not understanding your question. $\endgroup$
    – S2673
    Nov 20 '20 at 22:28
1
$\begingroup$

There are several functions that can be denoted as sigmoid functions, such as the logistic function and the hyperbolic tangent, given that they have an $S$-shaped curve. You can find more info about them in the related Wikipedia article.

However, when people use the term sigmoid function, they typically refer to the logistic function, which is a function of the form $$\sigma: \mathbb{R} \rightarrow [0, 1],$$ So, it accepts any real number and produces a number between $0$ and $1$, so you can pass any real number (as input) to the sigmoid.

The standard logistic function $\sigma$ is mathematically defined as follows

$$\sigma(x)=\frac{1}{1+e^{-x}}\in [0, 1]$$

Here is the graph of the logistic function. The $y$-axis contains the output values, i.e. $\sigma(x)$ for any $x \in \mathbb{R}$, while the $x$-axis contains the input values, i.e. $x \in \mathbb{R}$.

enter image description here

That function can be between 0 and 1 but my input values for the network are between -5 and 20. Someone told me that I need to scale the function so that it is in the range of -5 and 20 instead of 0 and 1. Is this true? Why?

No, in principle, you do not have to scale your inputs, for the reasons I have just explained. However, this does not mean that, in practice, it may not be a good idea to standardize or normalize your inputs before feeding them into the neural network. See the vanishing gradient problem (VGP) in this article (by J. Brownlee), or this one (by M. Nielsen), or in this video (by A. Ng). Moreover, you should probably actually use another activation function, such as the ReLU, which does not suffer from the VGP (which, nevertheless, should not occur, if your neural network does not contain more than a couple of hidden layers).

$\endgroup$
1
$\begingroup$

I presume when you say input you may be referring to the target values (the things you are trying to predict). If not, then some parts of your question might not make sense, like your proposal to apply a scaling.

In any case I would consider what the target distribution is before using a sigmoid and applying a scaling. The thing about a sigmoid is that the model has to be very confident in order to predict a value close to the boundaries. So in your case, the model would have to be really confident about something for it to output a -5, and it would have to be really confident of the opposite to output a +20. I would be surprised if this is what you are really looking for in this case.

Let's say your target distribution is uniform. Then your best bet is to use a linear activation, and maybe use max/min to clip it. Then it would behave a lot like a ReLU.

If I haven't quite nailed it with this answer, can you give more info on what you are predicting?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.