2022 Developer Survey is open! Take survey.

Questions tagged [sigmoid]

For questions about the sigmoid functions (in particular, the logistic functions) and the consequences of using them as activation functions in neural networks.

Filter by
Sorted by
Tagged with
0 votes
1 answer
37 views

Do the values over 0.5 mean my model classified the data as a "1" label and vice versa?

I am doing binary classification using an LSTM and my output layer is 1 neuron with a sigmoid function. My labels are either 0 or 1. ...
user avatar
0 votes
2 answers
33 views

Training a regression model on a set of values in 0-1 range to give 0-1 continual values

I have a textual dataset that has a set of real numbers as labels: L={0.0, 0.33, 0.5, 0.75, 1.0}, and I have a model that takes the text as input and has a Sigmoid output. If I train the model on this ...
user avatar
  • 123
0 votes
0 answers
132 views

How to predict using softmax having separate inputs and outputs?

I am new to Deep Learning. Having completed the coursera courses and read something from Deep Learning with Python, I am trying to implement one idea using DL. There is a number of user equipment (UE) ...
user avatar
  • 1
2 votes
1 answer
287 views

Why is there tanh(x)*sigmoid(x) in a LSTM cell?

CONTEXT I was wondering why there are sigmoid and tanh activation functions in an LSTM cell. My intuition was based on the flow of tanh(x)*sigmoid(x) and the ...
user avatar
2 votes
1 answer
143 views

How do sigmoid functions make it so that the prediction $\hat{y}$ indicates the probability that the observed value, $y$, is $1$?

I am currently studying the textbook Neural Networks and Deep Learning by Charu C. Aggarwal. Chapter 1.2.1.3 Choice of Activation and Loss Functions says the following: The choice of activation ...
user avatar
0 votes
1 answer
41 views

What is it about sigmoid activations in particular that allows for the keeping and forgetting of past information from different time scales?

My understanding is that normal recurrent neural networks (RNNs) are not good at keeping past information from different time scales. Furthermore, my understanding is that Gated RNNs, such as Long ...
user avatar
3 votes
1 answer
215 views

Why is it a problem if the outputs of an activation function are not zero-centered?

In this lecture, the professor says that one problem with the sigmoid function is that its outputs aren't zero-centered. Are the explanation provided by the professor regarding why this is bad is that ...
user avatar
  • 489
0 votes
0 answers
47 views

Why is the sigmoid function interpreted as a saturating firing rate of a neuron?

I've seen several people say that sigmoids are like a saturating firing rate of a neuron but I don't see how or why they interpret it as such. I especially don't see the relationship between a "...
user avatar
  • 489
1 vote
1 answer
833 views

Is it appropriate to use a softmax activation with a categorical crossentropy loss?

I have a binary classification problem where I have 2 classes. A sample is either class 1 or class 2 - For simplicity, lets say they are exclusive from one another so it is definitely one or the other....
user avatar
4 votes
0 answers
70 views

Why does sigmoid saturation prevent signal flow through the neuron?

As per these slides on page 35: Sigmoids saturate and kill gradients. when the neuron's activation saturates at either tail of 0 or 1, the gradient at these regions is almost zero. the gradient and ...
user avatar
  • 183
2 votes
0 answers
34 views

In the Binary Flower Pollination Algorithm (using the sigmoid function), is it possible that no feature is selected?

I'm trying to use the Binary Flower Pollination Algorithm (BFPA) for feature selection. In the BFPA, the sigmoid function is used to compute a binary vector that represents whether a feature is ...
user avatar
1 vote
2 answers
387 views

How to use sigmoid as transfer function when input is not (0,1) range in ANN?

I am building my first ANN from scratch. I know that I need a transfer function and I want to use the sigmoid function as my teacher recommended that. That function can be between 0 and 1, but my ...
user avatar
  • 111
1 vote
2 answers
71 views

Should I use additional empty category in some categorical problems?

I try to create autonomous car using keyboard data so this is a multi class classification problem. I have keys W,A,S and D. So I have four categories. My model should decide what key should be ...
user avatar
  • 173
0 votes
1 answer
382 views

Which kind of data does sigmoid kernel performance well?

While I was playing with some hyperparameters, I came to a wired situation. My dataset is IRIS dataset to be specific. SVM algorithm has some hyperparameters that we can tune, such as Kernels, and C ...
user avatar
3 votes
1 answer
82 views

Accuracy dropped when I ran the program the second time

I was following a tutorial about Feed-Forward Networks and wrote this code for a simple FFN : ...
user avatar
4 votes
1 answer
364 views

Neural network doesn't seem to converge with ReLU but it does with Sigmoid?

I'm not really sure if this is the sort of question to ask on here, since it is less of a general question about AI and more about the coding of it, however I thought it wouldn't fit on stack overflow....
user avatar
4 votes
1 answer
758 views

Can neural networks with a sigmoid as the activation function of the output layer approximate continuous functions?

Neural networks are commonly used for classification tasks, in fact from this post it seems like that's where they shine brightest. However, when we want to classify using neural networks, we often ...
user avatar
1 vote
0 answers
83 views

Do L2 regularization and input normalization depend on sigmoid activation functions?

Following the online courses with Andrew Ng, he talks about L2 regularization (a.k.a. weight decay) and input normalization. Now, the argument is that L2 regularization make the weights smaller, ...
user avatar
2 votes
1 answer
51 views

How can I train a neural network for another input set, without losing the learning of the previous input set?

I read this tutorial about backpropagation. So using this backpropagation we are training the neural network repeatedly for one input set, say [2,4], until we reach 100% accuracy of getting 1 as ...
user avatar
0 votes
1 answer
485 views

Network doesn't converge with ReLU or Leaky ReLU, but works well with sigmoid/tanh

I have these training data to separate, the classes are rather randomly scattered: My first attempt was using tf.nn.relu activation function, but output was stuck ...
user avatar
  • 1,153
2 votes
1 answer
288 views

Why will the sigmoid function be 1 (and 0), if we use a fully connected layer that produces a big enough positive (or negative, respectively) output?

I am using a fully connected neural network that uses a sigmoid activation function. If we feed a big enough input, the sigmoid function will finally become 1 or 0. Is there any solution to avoid this?...
user avatar
  • 121
9 votes
1 answer
2k views

What happens when I mix activation functions?

There are several activation functions, such as ReLU, sigmoid or $\tanh$. What happens when I mix activation functions? I recently found that Google has developed Swish activation function which is (...
user avatar
  • 93
1 vote
2 answers
144 views

Why do non-linear activation functions not require a specific non-linear relation between its inputs and outputs?

A linear activation function (or none at all) should only be used when the relation between input and output is linear. Why doesn't the same rule apply for other activation functions? For example, why ...
user avatar
1 vote
1 answer
804 views

Target values of 0.1 for 0 and 0.9 for 1 for sigmoid

I recently read an article about neural networks saying that, when using sigmoid as activation function, it's advised to use 0.1 as target value instead of 0, and 0.9 instead of 1. This was to avoid "...
user avatar
7 votes
1 answer
6k views

How is division by zero avoided when implementing back-propagation for a neural network with sigmoid at the output neuron?

I am building a neural network for which I am using the sigmoid function as the activation function for the single output neuron at the end. Since the sigmoid function is known to take any number and ...
user avatar
  • 73
1 vote
1 answer
613 views

How do I avoid the "math domain error" when the input to the log is zero in the objective function of a neural network?

I am implementing a neural network to train it on handwritten digits. Here is the cost function that I am implementing. $$J(\Theta)=-\frac{1}{m} \sum_{i=1}^{m} \sum_{k=1}^{K}\left[y_{k}^{(i)} \log \...
user avatar
8 votes
3 answers
2k views

Are ReLUs incapable of solving certain problems?

Background I've been interested in and reading about neural networks for several years, but I haven't gotten around to testing them out until recently. Both for fun and to increase my understanding, I ...
user avatar