Skip to main content

Questions tagged [activation-functions]

For questions related to the selection of and theory behind specific activation functions used in artificial networks.

Filter by
Sorted by
Tagged with
41 votes
5 answers
26k views

What is the purpose of an activation function in neural networks?

It is said that activation functions in neural networks help introduce non-linearity. What does this mean? What does non-linearity mean in this context? How does the introduction of this non-...
Mohsin's user avatar
  • 982
24 votes
3 answers
12k views

How to choose an activation function for the hidden layers?

I choose the activation function for the output layer depending on the output that I need and the properties of the activation function that I know. For example, I choose the sigmoid function when I'm ...
gvgramazio's user avatar
23 votes
1 answer
29k views

What are the advantages of ReLU vs Leaky ReLU and Parametric ReLU (if any)?

I think that the advantage of using Leaky ReLU instead of ReLU is that in this way we cannot have vanishing gradient. Parametric ReLU has the same advantage with the only difference that the slope of ...
gvgramazio's user avatar
19 votes
4 answers
6k views

What activation function does the human brain use?

Does the human brain use a specific activation function? I've tried doing some research, and as it's a threshold for whether the signal is sent through a neuron or not, it sounds a lot like ReLU. ...
mlman's user avatar
  • 301
18 votes
2 answers
4k views

Are softmax outputs of classifiers true probabilities?

BACKGROUND: The softmax function is the most common choice for an activation function for the last dense layer of a multiclass neural network classifier. The outputs of the softmax function have ...
Snehal Patel's user avatar
15 votes
4 answers
10k views

Why do activation functions need to be differentiable in the context of neural networks?

Why should an activation function of a neural network be differentiable? Is it strictly necessary or is it just advantageous?
user avatar
13 votes
1 answer
3k views

How exactly can ReLUs approximate non-linear and curved functions?

Currently, the most commonly used activation functions are ReLUs. So I answered this question What is the purpose of an activation function in neural networks? and, while writing the answer, it struck ...
user avatar
12 votes
3 answers
5k views

Why is the derivative of the activation functions in neural networks important?

I'm new to NN. I am trying to understand some of its foundations. One question that I have is: why the derivative of an activation function is important (not the function itself), and why it's the ...
Mary's user avatar
  • 973
12 votes
2 answers
880 views

What does it mean for a neuron in a neural network to be activated?

I just stumbled upon the concept of neuron coverage, which is the ratio of activated neurons and total neurons in a neural network. But what does it mean for a neuron to be "activated"? I know what ...
Leon's user avatar
  • 173
11 votes
2 answers
8k views

Why do we prefer ReLU over linear activation functions?

The ReLU activation function is defined as follows $$y = \operatorname{max}(0,x)$$ And the linear activation function is defined as follows $$y = x$$ The ReLU nonlinearity just clips the values ...
imflash217's user avatar
10 votes
1 answer
4k views

Why use ReLU over Leaky ReLU?

From my understanding a leaky ReLU attempts to address issues of vanishing gradients and nonzero-centeredness by keeping neurons that fire with a negative value alive. With just this info to go off of,...
John Brown's user avatar
10 votes
3 answers
2k views

Are ReLUs incapable of solving certain problems?

Background I've been interested in and reading about neural networks for several years, but I haven't gotten around to testing them out until recently. Both for fun and to increase my understanding, I ...
Benjamin Chambers's user avatar
9 votes
1 answer
2k views

When to use Tanh?

When and why would you not use Tanh? I just replaced ReLU with Tanh and my model trains about 2x faster, reaching 90% acc within 500 steps. While using ReLU it reached 90% acc in >1000 training ...
vxnuaj's user avatar
  • 93
9 votes
1 answer
4k views

What happens when I mix activation functions?

There are several activation functions, such as ReLU, sigmoid or $\tanh$. What happens when I mix activation functions? I recently found that Google has developed Swish activation function which is (...
JSChang's user avatar
  • 93
8 votes
1 answer
816 views

Why isn't the ElliotSig activation function widely used?

The Softsign (a.k.a. ElliotSig) activation function is really simple: $$ f(x) = \frac{x}{1+|x|} $$ It is bounded $[-1,1]$, has a first derivative, it is monotonic, and it is computationally ...
Pietro's user avatar
  • 183
8 votes
1 answer
3k views

Do all neurons in a layer have the same activation function?

I'm new to machine learning (so excuse my nomenclature), and not being a python developer, I decided to jump in at the deep (no pun intended) end writing my own framework in C++. In my current design, ...
lfgtm's user avatar
  • 230
8 votes
1 answer
1k views

What's the advantage of log_softmax over softmax?

Previously I have learned that the softmax as the output layer coupled with the log-likelihood cost function (the same as the ...
user1024's user avatar
  • 181
7 votes
1 answer
12k views

Why do ResNets avoid the vanishing gradient problem?

I read that, if we use the sigmoid or hyperbolic tangent activation functions in deep neural networks, we can have some problems with the vanishing of the gradient, and this is visible by the shapes ...
FraMan's user avatar
  • 199
7 votes
1 answer
166 views

What makes multi-layer neural networks able to perform nonlinear operations?

As I know, a single layer neural network can only do linear operations, but multilayered ones can. Also, I recently learned that finite matrices/tensors, which are used in many neural networks, can ...
Hyeonseo Yang's user avatar
6 votes
4 answers
3k views

Do neurons of a neural network model a linear relationship?

I'm certain that this is a very naive question, but I am just beginning to look more deeply at neural networks, having only used decision tree approaches in the past. Also, my formal mathematics ...
David Hoelzer's user avatar
6 votes
5 answers
1k views

Why can't the XOR linear inseparability problem be solved with one perceptron like this?

Consider a perceptron where $w_0=1$ and $w_1=1$: Now, suppose that we use the following activation function \begin{align} f(x)= \begin{cases} 1, \text{ if }x =1\\ 0, \text{ otherwise} \end{cases} \...
rahs's user avatar
  • 163
6 votes
1 answer
1k views

What is a "logit probability"?

DeepMind's paper "Mastering the game of Go without human knowledge" states in its "Methods" section on its "Neural network architecture" that the output layer of AlphaGo Zero's policy head is "A fully ...
sadakatsu's user avatar
  • 163
6 votes
1 answer
503 views

Smallest possible network to approximate the $sin$ function

The main goal is: Find the smallest possible neural network to approximate the $sin$ function. Moreover, I want to find a qualitative reason why this network is the smallest possible network. I have ...
JavAlex's user avatar
  • 75
6 votes
2 answers
2k views

In deep learning, is it possible to use discontinuous activation functions?

In deep learning, is it possible to use discontinuous activation functions (e.g. one with jump discontinuity)? (My guess: for example, ReLU is non-differentiable at a single point, but it still has a ...
Gyeonghoon Ko's user avatar
6 votes
1 answer
534 views

What is the mathematical definition of an activation function? [duplicate]

What is the mathematical definition of an activation function to be used in a neural network? So far I did not find a precise one, summarizing which criterions (e.g. monotonicity, differentiability, ...
user32649's user avatar
6 votes
4 answers
1k views

Is it suitable to find inverse of last layer's activation function and apply it on the target output?

I have a neural network with the following structure: I am expecting specific outputs from the neural network which are the target values for my training. Let's say the target values are 0.8 for the ...
Vikhyat Agarwal's user avatar
5 votes
2 answers
490 views

Why does the activation function for a hidden layer in a MLP have to be non-polynomial?

Across multiple pieces of literature describing MLPs or while describing the universal approximation theorem, the statement is very specific on the activation function being non-polynomial. Is there a ...
niil87's user avatar
  • 53
5 votes
1 answer
1k views

Which functions can be activation functions?

What are the required characteristics of an activation function (in a neural network)? Which functions can be activation functions? For example, which of the functions below can be used as an ...
mBabaee's user avatar
  • 51
5 votes
1 answer
1k views

Why is a softmax used rather than dividing each activation by the sum?

Just wondering why a softmax is typically used in practice on outputs of most neural nets rather than just summing the activations and dividing each activation by the sum. I know it's roughly the same ...
user8714896's user avatar
5 votes
2 answers
152 views

What should the range of the output layer be when performing classification?

I am working on a MLP neural networks, using supervised learning (2 classes and multi-class classification problems). For the hidden layers, I am using $\tanh$ (which produces an output in the range $[...
LVoltz's user avatar
  • 131
4 votes
2 answers
4k views

Why is no activation function used at the final layer of super-resolution models?

I'm trying to implement some image super-resolution models on medical images. After reading a set of papers, I found that none of the existing models use any activation function for the last layer. ...
Saeed's user avatar
  • 331
4 votes
1 answer
3k views

Is a non-linear activation function needed if we perform max-pooling after the convolution layer?

Is there any need to use a non-linear activation function (ReLU, LeakyReLU, Sigmoid, etc.) if the result of the convolution layer is passed through the sliding window max function, like max-pooling, ...
Kasia's user avatar
  • 303
4 votes
1 answer
151 views

If we had to choose between Uniform(0,1) and Uniform(-1,0), which one would you expect to work best and why?

I'm working with a fully connected neural network with input 32x32x3. The architecture includes a dense layer 32 + ReLu activation, then another dense layer 64 + ReLu Activation, followed by a ...
Miguel's user avatar
  • 43
4 votes
1 answer
3k views

Why is it a problem if the outputs of an activation function are not zero-centered?

In this lecture, the professor says that one problem with the sigmoid function is that its outputs aren't zero-centered. Are the explanation provided by the professor regarding why this is bad is that ...
Daviiid's user avatar
  • 575
4 votes
2 answers
195 views

What could be the problem when a neural network with four hidden layers with the sigmoid activation function is not learning?

I have a large set of data points describing mappings of binary vectors to real-valued outputs. I am using TensorFlow, and would like to train a model to predict these relationships. I used four ...
Aggraj Gupta's user avatar
4 votes
1 answer
4k views

How to constraint the output value of a neural network?

I am training a deep neural network. There is a constraint on the output value of the neural network (e.g. the output has to be between 0 and 180). I think some possible solutions are using sigmoid, ...
raemoii's user avatar
  • 41
4 votes
1 answer
1k views

Why should one ever use ReLU instead of PReLU?

To me, it seems that PReLU is strictly better than ReLU. It does not have the dying ReLU problem, it allows negative values and it has trainable parameters (which are computationally negligible to ...
algebruh's user avatar
4 votes
1 answer
193 views

What activation functions are currently popular?

I am not asking what activation function is better. I want to know what activation functions are more used in research or deployment. Also, are they used in combination? E.g., ReLU, ELUs, etc. I'd ...
Schach21's user avatar
  • 242
4 votes
2 answers
936 views

Is PReLU superfluous with respect to ReLU?

Why do people use the $PReLU$ activation? $PReLU[x] = ReLU[x] + ReLU[p*x]$ with the parameter $p$ typically being a small negative number. If a fully connected layer is followed by a at least two ...
Robert Nowak's user avatar
4 votes
1 answer
625 views

How do intermediate layers of a trained neural network look like?

Suppose I have a deep feed-forward neural network with sigmoid activation $\sigma$ already trained on a dataset $S$. Let's consider a training point $x_i \in S$. I want to analyze the entries of a ...
Alfred's user avatar
  • 165
4 votes
1 answer
138 views

Why is it important/beneficial for an activation function to be zero-meaned?

Conventionally, (although there are plenty of better options) it is being said that as the choice of activation function for hidden layers, tanh should be prefered ...
Onat Girit's user avatar
4 votes
0 answers
262 views

Why are most commonly used activation functions continuous?

I have come to notice that the most commonly used activation functions are continuous. Is there any specific reason behind this? Results such as this paper have worked on training networks with ...
ABIM's user avatar
  • 565
4 votes
0 answers
213 views

Has the logistic map ever been used as an activation function?

I find the logistic map absolutely fascinating. Both in itself (because I love fractal) and because it is observed in nature (see: https://www.youtube.com/watch?v=ovJcsL7vyrk). I'm wondering if anyone ...
ker2x's user avatar
  • 163
3 votes
2 answers
2k views

Can LSTM model use ReLU or LeakyReLU as the activation funtion?

Can LSTM model use ReLU or LeakyReLU as the activation funtion? If so, when should one use tanh and when should one use ReLU or LeakyReLU?
BlueSnake's user avatar
3 votes
3 answers
5k views

Why is there tanh(x)*sigmoid(x) in a LSTM cell?

CONTEXT I was wondering why there are sigmoid and tanh activation functions in an LSTM cell. My intuition was based on the flow of tanh(x)*sigmoid(x) and the ...
MASTER OF CODE's user avatar
3 votes
1 answer
110 views

Why aren't artificial derivatives used more often to solve the vanishing gradient problem?

While looking into the vanishing gradient problem, I came across a paper (https://ieeexplore.ieee.org/abstract/document/9336631) that used artificial derivatives in lieu of the real derivatives. For a ...
postnubilaphoebus's user avatar
3 votes
1 answer
1k views

Why is tanh a "smoothly" differentiable function?

The sigmoid, tanh, and ReLU are popular and useful activation functions in the literature. The following excerpt taken from p4 of Neural Networks and Neural Language Models says that ...
hanugm's user avatar
  • 3,890
3 votes
1 answer
793 views

How do two perceptrons produce different linear decision boundaries when learning?

I've learned that you can use two perceptrons to ultimately create a classifier for non-linearly separable data. I'm trying to understand how / if these two perceptrons converge to two different ...
ashar's user avatar
  • 39
3 votes
1 answer
988 views

Is the cube root function suitable as a n activation function?

I am trying to design a neural network on Python. Instead of the sigmoid function which has a limited range, I am thinking of using the cube root function which has the following graph: Is this ...
Vikhyat Agarwal's user avatar
3 votes
2 answers
866 views

What are the pros and cons of using sigmoid or softmax approach when dealing with 2 classes?

I know that when using Sigmoid, you only need 1 output neuron (binary classification) and for Softmax - it's 2 neurons (multiclass classification). But for performance improvement (if there is one), ...
Artūras Drūteika's user avatar