Skip to main content
Share Your Experience: Take the 2024 Developer Survey

Questions tagged [sigmoid]

For questions about the sigmoid functions (in particular, the logistic functions) and the consequences of using them as activation functions in neural networks.

Filter by
Sorted by
Tagged with
0 votes
0 answers
14 views

Any LMs that use tanh (generalization) instead of sigmoid within Attention?

Question is in the title. Posts such as this and this mention how this would be possible. I have some colleagues who have anecdotally heard of this being done on very small transformer models but I ...
naston's user avatar
  • 11
0 votes
0 answers
45 views

Find the relationship between data in this plot

Attached image. How would you find the relationship between independent variable (x) and dependent variable (y)? Is it linear or non-linear? What would the function looks like? P.S. I believe this is ...
DLCVIP007's user avatar
0 votes
0 answers
22 views

How does the relu nonlinearity allow for nonlinear polynomial transformations?

Since the neural network nonlinearities allow for nonlinear transformations that can stretch and squish the function, how can the ReLU activation function do this? I think for the sigmoid nonlinearity ...
Vivek Reddy's user avatar
2 votes
1 answer
60 views

If the output is 0.09, does this mean that the prediction is class 1 or 0?

I use a Keras EfficientNetB7 and transfer learning to solve a binary classification problem. I use tf.keras.layers.Dense(1, activation="sigmoid")(x) for ...
Doug's user avatar
  • 25
1 vote
1 answer
84 views

Approximate the chance of a number is prime using neural network

How to create a model that can give an output with a range of 0 to 1 with a sigmoid activation function where the value closer to 0 means the lesser chance that the input number is not prime and the ...
Muhammad Ikhwan Perwira's user avatar
7 votes
4 answers
2k views

What does "e" do in the Sigmoid Activation Function?

Within the Sigmoid Squishification function, f(x) = 1/(1 + e^(-x)) "e" is unnecessary, as it can be replaced by any other value that is not ...
Jake's user avatar
  • 81
0 votes
1 answer
47 views

How can a Regression based Neural Network learn class thresholds?

I understand that to solve multilabel classification problems, we can use the softmax activation function in the output layer of the neural network. The softmax function outputs probabilities of each ...
Dawood Ahmad's user avatar
0 votes
1 answer
105 views

Why is `SigmoidBinaryCrossEntropyLoss` in `DJL` implemented this way?

SigmoidBinaryCrossEntropyLoss implementation in DJL accepts two kinds of outputs from NNs: where sigmoid activation has already been applied. where raw NN output ...
src091's user avatar
  • 1
0 votes
1 answer
94 views

How does a sigmoid neuron act like a perceptron in this scenario?

I have been reading Michael Nielsen’s book online on his website at http://neuralnetworksanddeeplearning.com/chap1.html. I am struggling to understand the second exercise: When c approaches infinity, ...
QuantNoob's user avatar
0 votes
2 answers
165 views

Is this the correct way to backpropagate a Neural Network?

I am writing a Neural Network frorm scratch. Below is what I have right now, based off of the math that I think I understand. ...
user avatar
1 vote
1 answer
304 views

Does it make sense to provide a DQN with negative rewards for a network with relu and sigmoid activations?

The creation of negative rewards leads to the chance of Q-values being negative. However, networks with relu or sigmoid activations, just cannot predict negative values. This will lead to a case where ...
desert_ranger's user avatar
0 votes
1 answer
704 views

Do the values over 0.5 mean my model classified the data as a "1" label and vice versa?

I am doing binary classification using an LSTM and my output layer is 1 neuron with a sigmoid function. My labels are either 0 or 1. ...
Allen Ye's user avatar
0 votes
2 answers
2k views

Training a regression model on a set of values in 0-1 range to give 0-1 continual values

I have a textual dataset that has a set of real numbers as labels: L={0.0, 0.33, 0.5, 0.75, 1.0}, and I have a model that takes the text as input and has a Sigmoid output. If I train the model on this ...
Minions's user avatar
  • 123
3 votes
3 answers
5k views

Why is there tanh(x)*sigmoid(x) in a LSTM cell?

CONTEXT I was wondering why there are sigmoid and tanh activation functions in an LSTM cell. My intuition was based on the flow of tanh(x)*sigmoid(x) and the ...
MASTER OF CODE's user avatar
2 votes
1 answer
980 views

How do sigmoid functions make it so that the prediction $\hat{y}$ indicates the probability that the observed value, $y$, is $1$?

I am currently studying the textbook Neural Networks and Deep Learning by Charu C. Aggarwal. Chapter 1.2.1.3 Choice of Activation and Loss Functions says the following: The choice of activation ...
The Pointer's user avatar
0 votes
1 answer
55 views

What is it about sigmoid activations in particular that allows for the keeping and forgetting of past information from different time scales?

My understanding is that normal recurrent neural networks (RNNs) are not good at keeping past information from different time scales. Furthermore, my understanding is that Gated RNNs, such as Long ...
The Pointer's user avatar
4 votes
1 answer
3k views

Why is it a problem if the outputs of an activation function are not zero-centered?

In this lecture, the professor says that one problem with the sigmoid function is that its outputs aren't zero-centered. Are the explanation provided by the professor regarding why this is bad is that ...
Daviiid's user avatar
  • 575
0 votes
0 answers
204 views

Why is the sigmoid function interpreted as a saturating firing rate of a neuron?

I've seen several people say that sigmoids are like a saturating firing rate of a neuron but I don't see how or why they interpret it as such. I especially don't see the relationship between a "...
Daviiid's user avatar
  • 575
1 vote
1 answer
4k views

Is it appropriate to use a softmax activation with a categorical crossentropy loss?

I have a binary classification problem where I have 2 classes. A sample is either class 1 or class 2 - For simplicity, lets say they are exclusive from one another so it is definitely one or the other....
user9317212's user avatar
4 votes
0 answers
169 views

Why does sigmoid saturation prevent signal flow through the neuron?

As per these slides on page 35: Sigmoids saturate and kill gradients. when the neuron's activation saturates at either tail of 0 or 1, the gradient at these regions is almost zero. the gradient and ...
EEAH's user avatar
  • 193
2 votes
0 answers
44 views

In the Binary Flower Pollination Algorithm (using the sigmoid function), is it possible that no feature is selected?

I'm trying to use the Binary Flower Pollination Algorithm (BFPA) for feature selection. In the BFPA, the sigmoid function is used to compute a binary vector that represents whether a feature is ...
Adnan Hussein's user avatar
1 vote
2 answers
2k views

How to use sigmoid as transfer function when input is not (0,1) range in ANN?

I am building my first ANN from scratch. I know that I need a transfer function and I want to use the sigmoid function as my teacher recommended that. That function can be between 0 and 1, but my ...
J. Doe's user avatar
  • 111
1 vote
2 answers
78 views

Should I use additional empty category in some categorical problems?

I try to create autonomous car using keyboard data so this is a multi class classification problem. I have keys W,A,S and D. So I have four categories. My model should decide what key should be ...
user avatar
0 votes
1 answer
1k views

Which kind of data does sigmoid kernel performance well?

While I was playing with some hyperparameters, I came to a wired situation. My dataset is IRIS dataset to be specific. SVM algorithm has some hyperparameters that we can tune, such as Kernels, and C ...
Gooday2die's user avatar
3 votes
1 answer
101 views

Accuracy dropped when I ran the program the second time

I was following a tutorial about Feed-Forward Networks and wrote this code for a simple FFN : ...
Eeshaan Jain's user avatar
4 votes
1 answer
899 views

Neural network doesn't seem to converge with ReLU but it does with Sigmoid?

I'm not really sure if this is the sort of question to ask on here, since it is less of a general question about AI and more about the coding of it, however I thought it wouldn't fit on stack overflow....
finlay morrison's user avatar
4 votes
1 answer
1k views

Can neural networks with a sigmoid as the activation function of the output layer approximate continuous functions?

Neural networks are commonly used for classification tasks, in fact from this post it seems like that's where they shine brightest. However, when we want to classify using neural networks, we often ...
ABIM's user avatar
  • 565
1 vote
0 answers
183 views

Do L2 regularization and input normalization depend on sigmoid activation functions?

Following the online courses with Andrew Ng, he talks about L2 regularization (a.k.a. weight decay) and input normalization. Now, the argument is that L2 regularization make the weights smaller, ...
Ketil Malde's user avatar
2 votes
1 answer
58 views

How can I train a neural network for another input set, without losing the learning of the previous input set?

I read this tutorial about backpropagation. So using this backpropagation we are training the neural network repeatedly for one input set, say [2,4], until we reach 100% accuracy of getting 1 as ...
Tyrostir's user avatar
0 votes
1 answer
998 views

Network doesn't converge with ReLU or Leaky ReLU, but works well with sigmoid/tanh

I have these training data to separate, the classes are rather randomly scattered: My first attempt was using tf.nn.relu activation function, but output was stuck ...
Dan D.'s user avatar
  • 1,283
2 votes
1 answer
369 views

Why will the sigmoid function be 1 (and 0), if we use a fully connected layer that produces a big enough positive (or negative, respectively) output?

I am using a fully connected neural network that uses a sigmoid activation function. If we feed a big enough input, the sigmoid function will finally become 1 or 0. Is there any solution to avoid this?...
ou2105's user avatar
  • 121
9 votes
1 answer
4k views

What happens when I mix activation functions?

There are several activation functions, such as ReLU, sigmoid or $\tanh$. What happens when I mix activation functions? I recently found that Google has developed Swish activation function which is (...
JSChang's user avatar
  • 93
1 vote
2 answers
160 views

Why do non-linear activation functions not require a specific non-linear relation between its inputs and outputs?

A linear activation function (or none at all) should only be used when the relation between input and output is linear. Why doesn't the same rule apply for other activation functions? For example, why ...
Mr. Eivind's user avatar
1 vote
1 answer
1k views

Target values of 0.1 for 0 and 0.9 for 1 for sigmoid

I recently read an article about neural networks saying that, when using sigmoid as activation function, it's advised to use 0.1 as target value instead of 0, and 0.9 instead of 1. This was to avoid "...
Mr. Eivind's user avatar
7 votes
1 answer
8k views

How is division by zero avoided when implementing back-propagation for a neural network with sigmoid at the output neuron?

I am building a neural network for which I am using the sigmoid function as the activation function for the single output neuron at the end. Since the sigmoid function is known to take any number and ...
Dimitry's user avatar
  • 73
1 vote
1 answer
666 views

How do I avoid the "math domain error" when the input to the log is zero in the objective function of a neural network?

I am implementing a neural network to train it on handwritten digits. Here is the cost function that I am implementing. $$J(\Theta)=-\frac{1}{m} \sum_{i=1}^{m} \sum_{k=1}^{K}\left[y_{k}^{(i)} \log \...
Gokulakannan's user avatar
10 votes
3 answers
2k views

Are ReLUs incapable of solving certain problems?

Background I've been interested in and reading about neural networks for several years, but I haven't gotten around to testing them out until recently. Both for fun and to increase my understanding, I ...
Benjamin Chambers's user avatar