2
$\begingroup$

I am using a fully connected neural network that uses sigmoid activation function. If we feed a big enough weights the sigmoid function will finally become 1 or 0, is there any solution to avoid this?

Will this lead to classical sigmoid problems vanishing gradient or exploding gradient?

Improve this question
$\endgroup$
  • $\begingroup$ Increase precision to 256 bit if you are really desperate to use sigmoid. Else use different activation. $\endgroup$ – DuttaA Sep 11 '19 at 14:54
  • $\begingroup$ DuttaA I think I am stuck with the use of sigmoid(since I am looking for a probability ) but I wonder how to increase the precision to 256 what do you mean by that ? $\endgroup$ – ou2105 Sep 11 '19 at 15:15
  • 1
    $\begingroup$ Normal libraries use 64 bit precision for calculations. But if it is required you can increase precision thus avoiding a complete 0 gradient at the trade-off for memory and speed. But if you need probablity use softmax instead, and in the mid layers use relu. $\endgroup$ – DuttaA Sep 11 '19 at 15:55
  • $\begingroup$ The other problem is when using sigmoid in an FNN with more than 1 hidden layer is that backpropagation degrades and weights in the first layer will only be updated by small values that might not have a significant impact. Relu fixes this problem. I agree with @DuttaA use softmax if you want to get a probability. $\endgroup$ – SandMan Sep 11 '19 at 17:01
  • $\begingroup$ Can you elaborate on the task you're intending to solve with your network? As others have already said, ReLUs are the best option in intermediate layers and it's probable that you could do with logit (pre-sigmoid) outputs instead of going through the sigmoid for backprop purposes. $\endgroup$ – David Sep 11 '19 at 17:43
1
$\begingroup$

In general it's better to not use sigmoid function in any hidden layer. There are many other great options such as ReLU and ELU. However, if for any reason you have to use sigmoid-like function, then go with Tanh function, at least it has ~0 mean.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.