18 votes
Accepted

What does "e" do in the Sigmoid Activation Function?

The choice of $e$ is convenient when taking derivatives. Compare $\frac{d}{dx} \exp(x)$ to $\frac{d}{dx} a^x$ for any other $a > 0$.
Sycorax's user avatar
  • 452
5 votes

What does "e" do in the Sigmoid Activation Function?

If $d$ is a positive real number different from $1$, then $$d^{-x}=e^{-x\ln(d)}$$ So $d^{-x}$ is obtained from $e^{-x}$ by a horizontal shrink (when $\ln(d)>1$, that is $d>e$) or by a horizontal ...
Taladris's user avatar
  • 151
5 votes

What does "e" do in the Sigmoid Activation Function?

To add to other answers: Note that the usefulness of $e$ as the base is not limited to this particular case of sigmoid activation function. It is the go-to base in so many areas of mathematics because ...
JiK's user avatar
  • 151
3 votes

Why is there tanh(x)*sigmoid(x) in a LSTM cell?

I think a better way to understand LSTMs is by their purpose, instead of gradients and distributions. If you analyze the interactions of each gate with the cell state, you'll realize that LSTMs ...
Cyruno's user avatar
  • 31
3 votes
Accepted

Why is there tanh(x)*sigmoid(x) in a LSTM cell?

The tanh functions within the cell represent cell output or cell state. These are the values that either get passed on to other layers, or within the layer to the next time step. In theory, other ...
Neil Slater's user avatar
  • 32.1k
3 votes
Accepted

How do sigmoid functions make it so that the prediction $\hat{y}$ indicates the probability that the observed value, $y$, is $1$?

I am specifically asking about the probability that the value is 1 (that is, how sigmoid functions specifically check for this). They don't in general. In the quoted text, there is an explicit ...
Neil Slater's user avatar
  • 32.1k
3 votes
Accepted

What happens when I mix activation functions?

The general answer to the behavior of combining common activation functions is that the laws of calculus must be applied, specifically differential calculus, the results must be obtained through ...
Douglas Daseeco's user avatar
3 votes

Accuracy dropped when I ran the program the second time

It is common during the training of Neural Networks for accuracy to improve for a while and then get worse -- in general, This is caused by over-fitting. It's also fairly common for the Neural Network ...
Faizy's user avatar
  • 1,114
3 votes
Accepted

Why is it a problem if the outputs of an activation function are not zero-centered?

Yes, if the activation function of the network is not zero centered, $y = f(x^{T}w)$ is always positive or always negative. Thus, the output of a layer is always being moved to either the positive ...
Faizy's user avatar
  • 1,114
3 votes
Accepted

Can neural networks with a sigmoid as the activation function of the output layer approximate continuous functions?

As far as I know, the sigmoid is often used as the activation function of the output layer mainly because it is a convenient way of producing an output $p \in [0, 1]$, which can be interpreted as a ...
nbro's user avatar
  • 40.5k
3 votes

How can I train a neural network for another input set, without losing the learning of the previous input set?

Yes, this is actually a limitation known as catastrophic forgetting. A proposed way to deal with this is elastic weight consolidation that "remembers old tasks by selectively slowing down learning on ...
cookiemonster's user avatar
3 votes
Accepted

Network doesn't converge with ReLU or Leaky ReLU, but works well with sigmoid/tanh

I don't think it is dead ReLU units as a main cause, although they may be happening as part of the NN failing. The NN architecture is too complex for the given task (too deep, too many neurons) and ...
Neil Slater's user avatar
  • 32.1k
3 votes
Accepted

If the output is 0.09, does this mean that the prediction is class 1 or 0?

If you are using binary cross-entropy as loss function, the output of Sigmoid functions represent the probability of $\mathbb{P}[y=1]$. Hence, it means that $\mathbb{P}[y=0] = 1 - \mathbb{P}[y=1]$. ...
OmG's user avatar
  • 1,816
2 votes
Accepted

Are ReLUs incapable of solving certain problems?

There are a variety of possible things that could be wrong, but let me give you some potentially useful information. Neural networks with ReLU activation functions are Turing complete for a ...
lahwran's user avatar
  • 136
2 votes

Training a regression model on a set of values in 0-1 range to give 0-1 continual values

Many machine learning models used for regression will interpolate their predictions as you seem to want, and can return target values not seen in the training set. For example, basic linear regression ...
Neil Slater's user avatar
  • 32.1k
2 votes
Accepted

Training a regression model on a set of values in 0-1 range to give 0-1 continual values

As long as you train the model with a proper loss function for regression the model will learn to output any continuous values, not restricted to and most likely not exactly equal to the labels your ...
Edoardo Guerriero's user avatar
2 votes
Accepted

Does it make sense to provide a DQN with negative rewards for a network with relu and sigmoid activations?

A network with ReLU activation can predict negative values; we put ReLU between the hidden layers but return the output of the final layer without any activation function, or with a linear activation ...
Lee Reeves's user avatar
2 votes
Accepted

Target values of 0.1 for 0 and 0.9 for 1 for sigmoid

Derivative of the sigmoid curve is 0 when the output is 0 or 1 as you can see from the image above. The technique you are referring to is called label-smoothing which is used in various applications (...
Mark's user avatar
  • 106
2 votes

What does "e" do in the Sigmoid Activation Function?

$f$ is the unique function such that $f(0) = \frac{1}{2}$ and $f'(x) = f(x)(1 - f(x))$. Using $e$ is necessary to make sure the derivative takes this very simple form.
Charles Hudgins's user avatar
1 vote
Accepted

How can a Regression based Neural Network learn class thresholds?

First thing to notice, is that the assumptions on the target don't match the ones of multi-classifications: in particular, in multi-class classification, it's generally assumed that any other class ...
Alberto's user avatar
  • 1,915
1 vote

How does a sigmoid neuron act like a perceptron in this scenario?

When c approaches infinity, wouldn’t make the sigmoid function always output a value close to 1 whereas a perceptron can output 0 or 1. This is true only when the original value was positive. $e^{-cx}...
DKDK's user avatar
  • 329
1 vote
Accepted

Is this the correct way to backpropagate a Neural Network?

Your $d_2$ is the gradient used to update $w_2$, which is of course $\frac{dL}{dw_2}$. To compute this gradient, using your notation: $$ \frac{dL}{dw_2} = \frac{dL}{da_2}\frac{da_2}{dz_2}\frac{dz_2}{...
Martino's user avatar
  • 215
1 vote
Accepted

Do the values over 0.5 mean my model classified the data as a "1" label and vice versa?

Yes, the values over 0.5 mean the output should have "1" label. As I know with Keras you cannot set the optimal threshold, but you still can use the trick if you want, for example, the ...
CuCaRot's user avatar
  • 912
1 vote

Why is there tanh(x)*sigmoid(x) in a LSTM cell?

The purpose of the tanh and sigmoid functions in an LSTM (Long Short-Term Memory) network is to control the flow of information through the cell state, which is the "memory" of the network. ...
Suman's user avatar
  • 111
1 vote
Accepted

What is it about sigmoid activations in particular that allows for the keeping and forgetting of past information from different time scales?

It is not the sigmoid in particular. LSTMs and other memory-based recurrent networks are based on the idea of keeping an internal state that acts as a "canvas" in which the model can decide ...
m-pana's user avatar
  • 26
1 vote
Accepted

Is it appropriate to use a softmax activation with a categorical crossentropy loss?

Let's first recap the definition of the binary cross-entropy (BCE) and the categorical cross-entropy (CCE). Here's the BCE (equation 4.90 from this book) $$-\sum_{n=1}^{N}\left( t_{n} \ln y_{n}+\left(...
nbro's user avatar
  • 40.5k
1 vote

How to use sigmoid as transfer function when input is not (0,1) range in ANN?

I presume when you say input you may be referring to the target values (the things you are trying to predict). If not, then some parts of your question might not make sense, like your proposal to ...
Alexander Soare's user avatar
1 vote

How to use sigmoid as transfer function when input is not (0,1) range in ANN?

There are several functions that can be denoted as sigmoid functions, such as the logistic function and the hyperbolic tangent, given that they have an $S$-shaped curve. You can find more info about ...
nbro's user avatar
  • 40.5k
1 vote
Accepted

Should I use additional empty category in some categorical problems?

In short: yes, you must allow "do nothing" decision as a first level result. Your system must decide the action to be taken, including "do nothing" action. This is different to low ...
pasaba por aqui's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible