Questions tagged [sigmoid]
For questions about the sigmoid functions (in particular, the logistic functions) and the consequences of using them as activation functions in neural networks.
32
questions
0
votes
0
answers
41
views
Are there use cases of this "leaky hard sigmoid" activation function?
While trying to organize activation functions in a grid, this structure came up:
softplus -> relu -> leaky relu
sigmoid -> hard sigmoid -> unknown?
A ...
0
votes
0
answers
69
views
0
votes
1
answer
98
views
Why is `SigmoidBinaryCrossEntropyLoss` in `DJL` implemented this way?
SigmoidBinaryCrossEntropyLoss implementation in DJL accepts two kinds of outputs from NNs:
where sigmoid activation has already been applied.
where raw NN output ...
- 1
0
votes
1
answer
34
views
How does a sigmoid neuron act like a perceptron in this scenario?
I have been reading Michael Nielsen’s book online on his website at http://neuralnetworksanddeeplearning.com/chap1.html. I am struggling to understand the second exercise:
When c approaches infinity, ...
0
votes
2
answers
129
views
Is this the correct way to backpropagate a Neural Network?
I am writing a Neural Network frorm scratch. Below is what I have right now, based off of the math that I think I understand.
...
- 15
1
vote
1
answer
147
views
Does it make sense to provide a DQN with negative rewards for a network with relu and sigmoid activations?
The creation of negative rewards leads to the chance of Q-values being negative. However, networks with relu or sigmoid activations, just cannot predict negative values. This will lead to a case where ...
- 478
0
votes
1
answer
427
views
Do the values over 0.5 mean my model classified the data as a "1" label and vice versa?
I am doing binary classification using an LSTM and my output layer is 1 neuron with a sigmoid function. My labels are either 0 or 1.
...
- 3
0
votes
2
answers
774
views
Training a regression model on a set of values in 0-1 range to give 0-1 continual values
I have a textual dataset that has a set of real numbers as labels: L={0.0, 0.33, 0.5, 0.75, 1.0}, and I have a model that takes the text as input and has a Sigmoid output.
If I train the model on this ...
- 123
2
votes
3
answers
2k
views
Why is there tanh(x)*sigmoid(x) in a LSTM cell?
CONTEXT
I was wondering why there are sigmoid and tanh activation functions in an LSTM cell.
My intuition was based on the flow of tanh(x)*sigmoid(x)
and the ...
- 232
2
votes
1
answer
596
views
How do sigmoid functions make it so that the prediction $\hat{y}$ indicates the probability that the observed value, $y$, is $1$?
I am currently studying the textbook Neural Networks and Deep Learning by Charu C. Aggarwal. Chapter 1.2.1.3 Choice of Activation and Loss Functions says the following:
The choice of activation ...
- 477
0
votes
1
answer
46
views
What is it about sigmoid activations in particular that allows for the keeping and forgetting of past information from different time scales?
My understanding is that normal recurrent neural networks (RNNs) are not good at keeping past information from different time scales. Furthermore, my understanding is that Gated RNNs, such as Long ...
- 477
4
votes
1
answer
2k
views
Why is it a problem if the outputs of an activation function are not zero-centered?
In this lecture, the professor says that one problem with the sigmoid function is that its outputs aren't zero-centered. Are the explanation provided by the professor regarding why this is bad is that ...
- 519
0
votes
0
answers
128
views
Why is the sigmoid function interpreted as a saturating firing rate of a neuron?
I've seen several people say that sigmoids are like a saturating firing rate of a neuron but I don't see how or why they interpret it as such. I especially don't see the relationship between a "...
- 519
1
vote
1
answer
2k
views
Is it appropriate to use a softmax activation with a categorical crossentropy loss?
I have a binary classification problem where I have 2 classes. A sample is either class 1 or class 2 - For simplicity, lets say they are exclusive from one another so it is definitely one or the other....
- 151
4
votes
0
answers
126
views
Why does sigmoid saturation prevent signal flow through the neuron?
As per these slides on page 35:
Sigmoids saturate and kill gradients.
when the neuron's activation saturates at either tail of 0 or 1, the gradient at these regions is almost zero.
the gradient and ...
- 183
2
votes
0
answers
43
views
In the Binary Flower Pollination Algorithm (using the sigmoid function), is it possible that no feature is selected?
I'm trying to use the Binary Flower Pollination Algorithm (BFPA) for feature selection. In the BFPA, the sigmoid function is used to compute a binary vector that represents whether a feature is ...
1
vote
2
answers
1k
views
How to use sigmoid as transfer function when input is not (0,1) range in ANN?
I am building my first ANN from scratch. I know that I need a transfer function and I want to use the sigmoid function as my teacher recommended that. That function can be between 0 and 1, but my ...
- 111
1
vote
2
answers
74
views
Should I use additional empty category in some categorical problems?
I try to create autonomous car using keyboard data so this is a multi class classification problem. I have keys W,A,S and D. So I have four categories. My model should decide what key should be ...
user40943
0
votes
1
answer
774
views
Which kind of data does sigmoid kernel performance well?
While I was playing with some hyperparameters, I came to a wired situation. My dataset is IRIS dataset to be specific. SVM algorithm has some hyperparameters that we can tune, such as Kernels, and C ...
3
votes
1
answer
97
views
Accuracy dropped when I ran the program the second time
I was following a tutorial about Feed-Forward Networks and wrote this code for a simple FFN :
...
- 31
4
votes
1
answer
558
views
Neural network doesn't seem to converge with ReLU but it does with Sigmoid?
I'm not really sure if this is the sort of question to ask on here, since it is less of a general question about AI and more about the coding of it, however I thought it wouldn't fit on stack overflow....
4
votes
1
answer
1k
views
Can neural networks with a sigmoid as the activation function of the output layer approximate continuous functions?
Neural networks are commonly used for classification tasks, in fact from this post it seems like that's where they shine brightest.
However, when we want to classify using neural networks, we often ...
- 525
1
vote
0
answers
162
views
Do L2 regularization and input normalization depend on sigmoid activation functions?
Following the online courses with Andrew Ng, he talks about L2 regularization (a.k.a. weight decay) and input normalization. Now, the argument is that L2 regularization make the weights smaller, ...
- 224
2
votes
1
answer
56
views
How can I train a neural network for another input set, without losing the learning of the previous input set?
I read this tutorial about backpropagation.
So using this backpropagation we are training the neural network repeatedly for one input set, say [2,4], until we reach 100% accuracy of getting 1 as ...
- 29
0
votes
1
answer
729
views
Network doesn't converge with ReLU or Leaky ReLU, but works well with sigmoid/tanh
I have these training data to separate, the classes are rather randomly scattered:
My first attempt was using tf.nn.relu activation function, but output was stuck ...
- 1,283
2
votes
1
answer
354
views
Why will the sigmoid function be 1 (and 0), if we use a fully connected layer that produces a big enough positive (or negative, respectively) output?
I am using a fully connected neural network that uses a sigmoid activation function. If we feed a big enough input, the sigmoid function will finally become 1 or 0. Is there any solution to avoid this?...
- 121
9
votes
1
answer
3k
views
What happens when I mix activation functions?
There are several activation functions, such as ReLU, sigmoid or $\tanh$. What happens when I mix activation functions?
I recently found that Google has developed Swish activation function which is (...
- 93
1
vote
2
answers
153
views
Why do non-linear activation functions not require a specific non-linear relation between its inputs and outputs?
A linear activation function (or none at all) should only be used when the relation between input and output is linear. Why doesn't the same rule apply for other activation functions? For example, why ...
- 558
1
vote
1
answer
1k
views
Target values of 0.1 for 0 and 0.9 for 1 for sigmoid
I recently read an article about neural networks saying that, when using sigmoid as activation function, it's advised to use 0.1 as target value instead of 0, and 0.9 instead of 1. This was to avoid "...
- 558
7
votes
1
answer
7k
views
How is division by zero avoided when implementing back-propagation for a neural network with sigmoid at the output neuron?
I am building a neural network for which I am using the sigmoid function as the activation function for the single output neuron at the end. Since the sigmoid function is known to take any number and ...
- 73
1
vote
1
answer
641
views
How do I avoid the "math domain error" when the input to the log is zero in the objective function of a neural network?
I am implementing a neural network to train it on handwritten digits.
Here is the cost function that I am implementing.
$$J(\Theta)=-\frac{1}{m} \sum_{i=1}^{m} \sum_{k=1}^{K}\left[y_{k}^{(i)} \log \...
- 43
8
votes
3
answers
2k
views
Are ReLUs incapable of solving certain problems?
Background
I've been interested in and reading about neural networks for several years, but I haven't gotten around to testing them out until recently.
Both for fun and to increase my understanding, I ...
