Questions tagged [sigmoid]

For questions about the sigmoid functions (in particular, the logistic functions) and the consequences of using them as activation functions in neural networks.

Filter by
Sorted by
Tagged with
10
votes
1answer
2k views

What happens when I mix activation functions?

There are several activation functions, such as ReLU, sigmoid or $\tanh$. What happens when I mix activation functions? I recently found that Google has developed Swish activation function which is (...
9
votes
3answers
1k views

Are ReLUs incapable of solving certain problems?

Background I've been interested in and reading about neural networks for several years, but I haven't gotten around to testing them out until recently. Both for fun and to increase my understanding, I ...
8
votes
1answer
653 views

Can neural networks with a sigmoid as the activation function of the output layer approximate continuous functions?

Neural networks are commonly used for classification tasks, in fact from this post it seems like that's where they shine brightest. However, when we want to classify using neural networks, we often ...
3
votes
1answer
193 views

Neural network doesn't seem to converge with ReLU but it does with Sigmoid?

I'm not really sure if this is the sort of question to ask on here, since it is less of a general question about AI and more about the coding of it, however I thought it wouldn't fit on stack overflow....
3
votes
0answers
42 views

Why does sigmoid saturation prevent signal flow through the neuron?

As per these slides on page 35: Sigmoids saturate and kill gradients. when the neuron's activation saturates at either tail of 0 or 1, the gradient at these regions is almost zero. the gradient and ...
2
votes
1answer
66 views

Accuracy dropped when I ran the program the second time

I was following a tutorial about Feed Forward Networks and wrote this code for a simple FFN : ...
2
votes
1answer
49 views

How can I train a neural network for another input set, without losing the learning of the previous input set?

I read this tutorial about backpropagation. So using this backpropagation we are training the neural network repeatedly for one input set, say [2,4], until we reach 100% accuracy of getting 1 as ...
2
votes
1answer
62 views

How do sigmoid functions make it so that the prediction $\hat{y}$ indicates the probability that the observed value, $y$, is $1$?

I am currently studying the textbook Neural Networks and Deep Learning by Charu C. Aggarwal. Chapter 1.2.1.3 Choice of Activation and Loss Functions says the following: The choice of activation ...
2
votes
1answer
147 views

Why is it a problem if the outputs of an activation function are not zero-centered?

In this lecture, the professor says that one problem with the sigmoid function is that its outputs aren't zero-centered. Are the explanation provided by the professor regarding why this is bad is that ...
2
votes
0answers
34 views

In the Binary Flower Pollination Algorithm (using the sigmoid function), is it possible that no feature is selected?

I'm trying to use the Binary Flower Pollination Algorithm (BFPA) for feature selection. In the BFPA, the sigmoid function is used to compute a binary vector that represents whether a feature is ...
2
votes
2answers
143 views

Why do non-linear activation functions not require a specific non-linear relation between its inputs and outputs?

A linear activation function (or none at all) should only be used when the relation between input and output is linear. Why doesn't the same rule apply for other activation functions? For example, why ...
1
vote
2answers
61 views

Should I use additional empty category in some categorical problems?

I try to create autonomous car using keyboard data so this is a multi class classification problem. I have keys W,A,S and D. So I have four categories. My model should decide what key should be ...
1
vote
2answers
184 views

How to use sigmoid as transfer function when input is not (0,1) range in ANN?

I am building my first ANN from scratch. I know that I need a transfer function and I want to use the sigmoid function as my teacher recommended that. That function can be between 0 and 1, but my ...
1
vote
1answer
318 views

Is it appropriate to use a softmax activation with a categorical crossentropy loss?

I have a binary classification problem where I have 2 classes. A sample is either class 1 or class 2 - For simplicity, lets say they are exclusive from one another so it is definitely one or the other....
1
vote
0answers
49 views

Do L2 regularization and input normalization depend on sigmoid activation functions?

Following the online courses with Andrew Ng, he talks about L2 regularization (a.k.a. weight decay) and input normalization. Now, the argument is that L2 regularization make the weights smaller, ...
0
votes
1answer
639 views

Target values of 0.1 for 0 and 0.9 for 1 for sigmoid

I recently read an article about neural networks saying that, when using sigmoid as activation function, it's advised to use 0.1 as target value instead of 0, and 0.9 instead of 1. This was to avoid "...
0
votes
1answer
36 views

What is it about sigmoid activations in particular that allows for the keeping and forgetting of past information from different time scales?

My understanding is that normal recurrent neural networks (RNNs) are not good at keeping past information from different time scales. Furthermore, my understanding is that Gated RNNs, such as Long ...
0
votes
1answer
326 views

Network doesn't converge with ReLU or Leaky ReLU, but works well with sigmoid/tanh

I have these training data to separate, the classes are rather randomly scattered: My first attempt was using tf.nn.relu activation function, but output was stuck ...
0
votes
0answers
29 views

Why is the sigmoid function interpreted as a saturating firing rate of a neuron?

I've seen several people say that sigmoids are like a saturating firing rate of a neuron but I don't see how or why they interpret it as such. I especially don't see the relationship between a "...
0
votes
1answer
174 views

Which kind of data does sigmoid kernel performance well?

While I was playing with some hyperparameters, I came to a wired situation. My dataset is IRIS dataset to be specific. SVM algorithm has some hyperparameters that we can tune, such as Kernels, and C ...
0
votes
1answer
82 views

Is my backpropagation code correct? [closed]

I am trying to implement the back-propagation algorithm for the following neural network. ...