# Questions tagged [activation-functions]

For questions related to the selection of and theory behind specific activation functions used in artificial networks.

167 questions
Filter by
Sorted by
Tagged with
2k views

### When to use Tanh?

When and why would you not use Tanh? I just replaced ReLU with Tanh and my model trains about 2x faster, reaching 90% acc within 500 steps. While using ReLU it reached 90% acc in >1000 training ...
• 83
44 views

### Xavier vs He initialization with tanh

I'm a student and in the lecture, I learned that He initialization is better than Xavier if you use ReLU activation function. In addition, I also learned that Xavier initialization is better than He ...
1 vote
66 views

### Activation function intuition question

I just want to verify the my intuition of why activation functions are necessary. For this example lets consider a network that classifies numbers 0-9. A network WITHOUT an activation function will be ...
• 95
22 views

### How does the relu nonlinearity allow for nonlinear polynomial transformations?

Since the neural network nonlinearities allow for nonlinear transformations that can stretch and squish the function, how can the ReLU activation function do this? I think for the sigmoid nonlinearity ...
148 views

### If we had to choose between Uniform(0,1) and Uniform(-1,0), which one would you expect to work best and why?

I'm working with a fully connected neural network with input 32x32x3. The architecture includes a dense layer 32 + ReLu activation, then another dense layer 64 + ReLu Activation, followed by a ...
• 43
33 views

### Multi-Task VAE, Decoder Activation Functions?

I'm working on a Multi-Task VAE with one Encoder and two Decoders. The input consists of a vector with parameters which describe a design of a fluid system. The goal is to reconstruct the parameters ...
78 views

### Why does my activation function cause NaNs?

I implemented the following activation function: $$\sigma(x) = x^{1/3},$$ which returns NaNs after some epochs. I think this is due to the derivative exploding close to $0$. To fix this issue, I ...
• 101
45 views

### Is It Reasonable to Get NaN Outputs With Identity Activation Functions

I made my own Neural Network from scratch in unity with C# and I am using it as DQN. I set up my network which has 4 layers: 9 input values, 20 nodes in the second layer, 15 nodes in the third layer, ...
• 25
1 vote
36 views

### How do we get from types of activation functions to fitting lines to our data?

I'm completely new to AI and admittedly have never been good at math (also please excuse me if I use the wrong terminology). Despite this, I'm trying wrap my head around activation functions and how ...
• 181
542 views

### LUT-Based Sigmoid and Tanh Activation-Functions in Integer Quantized Networks

I want to understand how activation functions, specifically tanh and sigmoid, are used in int8 quantized neural networks. Even more specific, I want to understand a Look-up-Table based approach. My ...
57 views

### What is actually tuned during prompt engineering of autoregressive LLMs like GPT?

There are a lot of sites going over prompt engineering, but I don't see them explaining what is actually being changed. Is it hidden activation layers being tuned?
• 121
4k views

### Why use ReLU over Leaky ReLU?

From my understanding a leaky ReLU attempts to address issues of vanishing gradients and nonzero-centeredness by keeping neurons that fire with a negative value alive. With just this info to go off of,...
• 103
95 views

### Difference in gradient calculation for the last layer activation in neural networks

I'm currently working on implementing a neural network using the sigmoid activation function and the binary cross-entropy cost function. In my implementation, I've noticed that the gradient ...
1k views

### How do we determine the slope for leakyrelu activation function?

I am using LeakyReLU activation function in my architecture. We know that the default slope value is 1e-2. I want to understand ...
2k views

### Can LSTM model use ReLU or LeakyReLU as the activation funtion?

Can LSTM model use ReLU or LeakyReLU as the activation funtion? If so, when should one use tanh and when should one use ReLU or LeakyReLU?
1 vote
26 views

### Can an activation function with large derivative cause exploding gradient?

The maximum derivative of most of the currently existing activation functions is around 1. Can an activation function with derivatives higher than 1, say 1000 (a), cause exploding gradient problem? ...
• 11
483 views

### Why do activation functions in neural networks have to be non-polynomial to approximate any function?

Can someone give me the main idea of the paper Multilayer Feedforward Networks With a Nonpolynomial Activation Function Can Approximate Any Function? I'm having trouble understanding it.
4k views

### Are softmax outputs of classifiers true probabilities?

BACKGROUND: The softmax function is the most common choice for an activation function for the last dense layer of a multiclass neural network classifier. The outputs of the softmax function have ...
• 952
109 views

### Why aren't artificial derivatives used more often to solve the vanishing gradient problem?

While looking into the vanishing gradient problem, I came across a paper (https://ieeexplore.ieee.org/abstract/document/9336631) that used artificial derivatives in lieu of the real derivatives. For a ...
62 views

### Why is the derivative of activation function all positive?

All the activation functions I see have positive derivatives. Will negative ReLU work as well as its positive counterpart or will it lead to instability?
• 101
1 vote
142 views

### Neural network and logical gates

I have a network witch consist of two fully connected layers (without bias) and a ReLu function in between. The network input is two binary numbers, and the output should be the a logical gate result: ...
• 131
705 views

### Why cannot linear activation functions be used to approximate any function?

In neural networks we use nonlinear activation functions such as sigmoid, ReLU, etc. Using a combination of these functions (with required scaling and shifting), we manage to estimate any nonlinear ...
237 views

### Why and when do we use ReLU over tanh activation function?

I was reading LeCun Efficient Backprop and the author repeated stressed the importance of average the input patterns at 0 and thus justified the usage of tanh sigmoid. But if tanh is good then how ...
77 views

### Where does the "rectified" in ReLU come from?

ReLU stands for Rectified Linear Unit. Linear Unit, I understand, since the function is piecewise linear. But what does rectified mean? I looked up the definition and it said: denoting an electric ...
• 123
1 vote
236 views

### Why are SVMs / Softmax classifiers considered linear while neural networks are non-linear?

My understanding is that neural networks are definitely not linear classifiers, as the point of functions like ReLU is to introduce non-linearity. However, here's where my understanding starts to ...
• 153
874 views

### Is there any way to train a regression model with negative values that is more stable?

I have a regression model where my target values contain roughly 60% negative values and 40% positive values. My model architecture includes a robert-large, 1 linear layer. I trained it after 1 epoch, ...
136 views

### Why is it important/beneficial for an activation function to be zero-meaned?

Conventionally, (although there are plenty of better options) it is being said that as the choice of activation function for hidden layers, tanh should be prefered ...
231 views

### Should the range of target values match the range of activation function used in the output layer?

Suppose I use a tansig activation function in the output layer of an artificial neural network giving me outputs in the range $[-1,1]$ and my model is applied to a binary classification problem, ...
1 vote
250 views

### How to define a custom layer in Pytorch [closed]

I am new to PyTorch and seeking your help regarding a problem I have. I need to add a costume layer to a NN in training phase. Please see the figure which shows a simple DNN with the custom layer. NN ...
• 11
131 views

### What are the advantages and disadvantages of higher order neuron activation functions?

I've been reading about different types of neurons that the traditional linear one. One example I came across is the Sigma-Pi neuron, where the activation function includes higher order terms, such as ...
1 vote
2k views

### Using "softmax" (non-linear) vs "linear" activation function in Deep Reinforcement Learning

I am following the tutorial in this video: https://youtu.be/cO5g5qLrLSo which implements deep reinforcement learning (DQN) to balance cart pole in OpenAI default environment. The DQN model looks like ...
• 111
413 views

### Are there any scale invariant activation functions that outputs probability distribution?

Softmax activation function is used to convert any random vector into a probability distribution. So, it is generally used as an activation function in the last layer of deep neural networks that are ...
• 3,880
218 views

### Are any non-injective activation functions used?

All activation functions I know of are injective, which I think makes sense. But are there cases where non-injective activations can be useful?
• 133
1 vote
1k views

### How to output an integer/discrete number n with a single output neuron?

Say I have a game with 4 base actions [left, right, up, down] and then a value n, which determines how many times the chosen action is repeated. For example, action = left, n = 3 -> go left 3 times....
5k views

### Why is there tanh(x)*sigmoid(x) in a LSTM cell?

CONTEXT I was wondering why there are sigmoid and tanh activation functions in an LSTM cell. My intuition was based on the flow of tanh(x)*sigmoid(x) and the ...
476 views

### Why does the activation function for a hidden layer in a MLP have to be non-polynomial?

Across multiple pieces of literature describing MLPs or while describing the universal approximation theorem, the statement is very specific on the activation function being non-polynomial. Is there a ...
• 53
1 vote
3k views

### Why identity function is generally treated as an activation function?

It is known that the primary purpose of activation functions, used in neural networks, is to introduce non-linearity. Then how can the linear activation function, especially the identity function, be ...
• 3,880
1k views

### Why should one ever use ReLU instead of PReLU?

To me, it seems that PReLU is strictly better than ReLU. It does not have the dying ReLU problem, it allows negative values and it has trainable parameters (which are computationally negligible to ...
• 41
346 views

### Is it possible to use Softmax as an activation function for actor (policy) network in TD3 or SAC Reinforcement learning algorithms?

As I understand from literature, normally, the last activation in an actor (policy) network in TD3 and SAC algorithms is a Tanh function, which is scaled by a certain limit. My action vector is ...
• 101
1k views

### Does the output layer in a deep neural network need an activation function?

I have enrolled in a course that uses only one hidden layer, and that is the only layer that has activation functions. The model can be visualized as follows: and here is a PyTorch implementation: <...
150 views

### What is meant by "well-behaved gradient" in this context?

Consider the following statement (from the paper Generative Adversarial Nets) about the success of discriminative models So far, the most striking successes in deep learning have involved ...
• 3,880
610 views

### What does "linear unit" mean in the names of activation functions?

Activation functions, in neural networks, are used to introduce non-linearity. Many activation functions that are used in neural networks have the term "Linear Unit" in their full form. &...
• 3,880
489 views

### Is my understanding on "smooth approximation" correct?

Consider the following details regarding Softplus activation function $$\text{Softplus}(x) = \dfrac{\log(1+e^{\beta x})}{\beta}$$ SoftPlus is a smooth approximation to the ReLU function and can be ...
• 3,880
255 views

### Using a rectified Tanh to train a CNN?

I have been experimenting with activation functions on CNN, and it occurred to me to use a rectified tanh function. So that is basically if z > 0 tanh(z) else 0. ...
• 21
1 vote
76 views

### Is the main difference between the logistic regression and the perceptron the activation function they use?

I went through a Stats StackExchange's post about the difference between logistic regression and perceptron, which is too long to get the key point. I'd like to consider the question in terms of the ...
• 221
1k views

### Why is tanh a "smoothly" differentiable function?

The sigmoid, tanh, and ReLU are popular and useful activation functions in the literature. The following excerpt taken from p4 of Neural Networks and Neural Language Models says that ...
• 3,880
1 vote
672 views

### An explanation involving the sign activation, its affect on the loss function, and the perceptron and perceptron criterion: what is this saying?

I am currently studying the textbook Neural Networks and Deep Learning by Charu C. Aggarwal. Chapter 1.2.1.3 Choice of Activation and Loss Functions says the following: The classical activation ...
• 569
1 vote
39 views

### Are there guiding principles as to which activation functions suit a given RL algorithm?

Are there rules of thumb as to which activation functions work well (or which one would not) on the policy and value network of a class of RL algorithms? For hidden layers and for the output layer. ...
• 531
462 views

### Where does the so-called 'loss' / 'loss function' fit into the idea of a perceptron / artificial neuron (as presented in the figure)?

I am currently studying the textbook Neural Networks and Deep Learning by Charu C. Aggarwal. Chapter 1.2.1.3 Choice of Activation and Loss Functions presents the following figure: $\overline{X}$ is ...
• 569