Skip to main content
OverflowAI is here! AI power for your Stack Overflow for Teams knowledge community. Learn more

Questions tagged [activation-functions]

For questions related to the selection of and theory behind specific activation functions used in artificial networks.

Filter by
Sorted by
Tagged with
8 votes
1 answer
2k views

When to use Tanh?

When and why would you not use Tanh? I just replaced ReLU with Tanh and my model trains about 2x faster, reaching 90% acc within 500 steps. While using ReLU it reached 90% acc in >1000 training ...
vxnuaj's user avatar
  • 83
0 votes
1 answer
44 views

Xavier vs He initialization with tanh

I'm a student and in the lecture, I learned that He initialization is better than Xavier if you use ReLU activation function. In addition, I also learned that Xavier initialization is better than He ...
COTHE's user avatar
  • 5
1 vote
1 answer
66 views

Activation function intuition question

I just want to verify the my intuition of why activation functions are necessary. For this example lets consider a network that classifies numbers 0-9. A network WITHOUT an activation function will be ...
Stef's user avatar
  • 95
0 votes
0 answers
22 views

How does the relu nonlinearity allow for nonlinear polynomial transformations?

Since the neural network nonlinearities allow for nonlinear transformations that can stretch and squish the function, how can the ReLU activation function do this? I think for the sigmoid nonlinearity ...
Vivek Reddy's user avatar
4 votes
1 answer
148 views

If we had to choose between Uniform(0,1) and Uniform(-1,0), which one would you expect to work best and why?

I'm working with a fully connected neural network with input 32x32x3. The architecture includes a dense layer 32 + ReLu activation, then another dense layer 64 + ReLu Activation, followed by a ...
Miguel's user avatar
  • 43
0 votes
0 answers
33 views

Multi-Task VAE, Decoder Activation Functions?

I'm working on a Multi-Task VAE with one Encoder and two Decoders. The input consists of a vector with parameters which describe a design of a fluid system. The goal is to reconstruct the parameters ...
tekay's user avatar
  • 1
0 votes
1 answer
78 views

Why does my activation function cause NaNs?

I implemented the following activation function: $$\sigma(x) = x^{1/3},$$ which returns NaNs after some epochs. I think this is due to the derivative exploding close to $0$. To fix this issue, I ...
user572780's user avatar
0 votes
1 answer
45 views

Is It Reasonable to Get NaN Outputs With Identity Activation Functions

I made my own Neural Network from scratch in unity with C# and I am using it as DQN. I set up my network which has 4 layers: 9 input values, 20 nodes in the second layer, 15 nodes in the third layer, ...
Ege's user avatar
  • 25
1 vote
1 answer
36 views

How do we get from types of activation functions to fitting lines to our data?

I'm completely new to AI and admittedly have never been good at math (also please excuse me if I use the wrong terminology). Despite this, I'm trying wrap my head around activation functions and how ...
Garrett's user avatar
  • 181
0 votes
2 answers
542 views

LUT-Based Sigmoid and Tanh Activation-Functions in Integer Quantized Networks

I want to understand how activation functions, specifically tanh and sigmoid, are used in int8 quantized neural networks. Even more specific, I want to understand a Look-up-Table based approach. My ...
Necrotos's user avatar
0 votes
0 answers
57 views

What is actually tuned during prompt engineering of autoregressive LLMs like GPT?

There are a lot of sites going over prompt engineering, but I don't see them explaining what is actually being changed. Is it hidden activation layers being tuned?
user14094230's user avatar
10 votes
1 answer
4k views

Why use ReLU over Leaky ReLU?

From my understanding a leaky ReLU attempts to address issues of vanishing gradients and nonzero-centeredness by keeping neurons that fire with a negative value alive. With just this info to go off of,...
John Brown's user avatar
0 votes
1 answer
95 views

Difference in gradient calculation for the last layer activation in neural networks

I'm currently working on implementing a neural network using the sigmoid activation function and the binary cross-entropy cost function. In my implementation, I've noticed that the gradient ...
gratus richard's user avatar
2 votes
2 answers
1k views

How do we determine the slope for leakyrelu activation function?

I am using LeakyReLU activation function in my architecture. We know that the default slope value is 1e-2. I want to understand ...
Rohit Singh's user avatar
3 votes
2 answers
2k views

Can LSTM model use ReLU or LeakyReLU as the activation funtion?

Can LSTM model use ReLU or LeakyReLU as the activation funtion? If so, when should one use tanh and when should one use ReLU or LeakyReLU?
BlueSnake's user avatar
1 vote
0 answers
26 views

Can an activation function with large derivative cause exploding gradient?

The maximum derivative of most of the currently existing activation functions is around 1. Can an activation function with derivatives higher than 1, say 1000 (a), cause exploding gradient problem? ...
JGM's user avatar
  • 11
2 votes
1 answer
483 views

Why do activation functions in neural networks have to be non-polynomial to approximate any function?

Can someone give me the main idea of the paper Multilayer Feedforward Networks With a Nonpolynomial Activation Function Can Approximate Any Function? I'm having trouble understanding it.
Alicia Chi's user avatar
18 votes
2 answers
4k views

Are softmax outputs of classifiers true probabilities?

BACKGROUND: The softmax function is the most common choice for an activation function for the last dense layer of a multiclass neural network classifier. The outputs of the softmax function have ...
Snehal Patel's user avatar
3 votes
1 answer
109 views

Why aren't artificial derivatives used more often to solve the vanishing gradient problem?

While looking into the vanishing gradient problem, I came across a paper (https://ieeexplore.ieee.org/abstract/document/9336631) that used artificial derivatives in lieu of the real derivatives. For a ...
postnubilaphoebus's user avatar
0 votes
1 answer
62 views

Why is the derivative of activation function all positive?

All the activation functions I see have positive derivatives. Will negative ReLU work as well as its positive counterpart or will it lead to instability?
user33869's user avatar
  • 101
1 vote
0 answers
142 views

Neural network and logical gates

I have a network witch consist of two fully connected layers (without bias) and a ReLu function in between. The network input is two binary numbers, and the output should be the a logical gate result: ...
Daniel's user avatar
  • 131
2 votes
1 answer
705 views

Why cannot linear activation functions be used to approximate any function?

In neural networks we use nonlinear activation functions such as sigmoid, ReLU, etc. Using a combination of these functions (with required scaling and shifting), we manage to estimate any nonlinear ...
levitatmas's user avatar
2 votes
1 answer
237 views

Why and when do we use ReLU over tanh activation function?

I was reading LeCun Efficient Backprop and the author repeated stressed the importance of average the input patterns at 0 and thus justified the usage of tanh sigmoid. But if tanh is good then how ...
Struggling_In_Final's user avatar
0 votes
1 answer
77 views

Where does the "rectified" in ReLU come from?

ReLU stands for Rectified Linear Unit. Linear Unit, I understand, since the function is piecewise linear. But what does rectified mean? I looked up the definition and it said: denoting an electric ...
a6623's user avatar
  • 123
1 vote
1 answer
236 views

Why are SVMs / Softmax classifiers considered linear while neural networks are non-linear?

My understanding is that neural networks are definitely not linear classifiers, as the point of functions like ReLU is to introduce non-linearity. However, here's where my understanding starts to ...
Foobar's user avatar
  • 153
0 votes
1 answer
874 views

Is there any way to train a regression model with negative values that is more stable?

I have a regression model where my target values contain roughly 60% negative values and 40% positive values. My model architecture includes a robert-large, 1 linear layer. I trained it after 1 epoch, ...
Việt Nguyễn's user avatar
4 votes
1 answer
136 views

Why is it important/beneficial for an activation function to be zero-meaned?

Conventionally, (although there are plenty of better options) it is being said that as the choice of activation function for hidden layers, tanh should be prefered ...
Onat Girit's user avatar
0 votes
2 answers
231 views

Should the range of target values match the range of activation function used in the output layer?

Suppose I use a tansig activation function in the output layer of an artificial neural network giving me outputs in the range $[-1,1]$ and my model is applied to a binary classification problem, ...
Rowan Barua's user avatar
1 vote
0 answers
250 views

How to define a custom layer in Pytorch [closed]

I am new to PyTorch and seeking your help regarding a problem I have. I need to add a costume layer to a NN in training phase. Please see the figure which shows a simple DNN with the custom layer. NN ...
zstr's user avatar
  • 11
0 votes
0 answers
131 views

What are the advantages and disadvantages of higher order neuron activation functions?

I've been reading about different types of neurons that the traditional linear one. One example I came across is the Sigma-Pi neuron, where the activation function includes higher order terms, such as ...
burn_burn_55's user avatar
1 vote
1 answer
2k views

Using "softmax" (non-linear) vs "linear" activation function in Deep Reinforcement Learning

I am following the tutorial in this video: https://youtu.be/cO5g5qLrLSo which implements deep reinforcement learning (DQN) to balance cart pole in OpenAI default environment. The DQN model looks like ...
Somdip Dey's user avatar
0 votes
1 answer
413 views

Are there any scale invariant activation functions that outputs probability distribution?

Softmax activation function is used to convert any random vector into a probability distribution. So, it is generally used as an activation function in the last layer of deep neural networks that are ...
hanugm's user avatar
  • 3,880
-1 votes
1 answer
218 views

Are any non-injective activation functions used?

All activation functions I know of are injective, which I think makes sense. But are there cases where non-injective activations can be useful?
Moritz Groß's user avatar
1 vote
1 answer
1k views

How to output an integer/discrete number n with a single output neuron?

Say I have a game with 4 base actions [left, right, up, down] and then a value n, which determines how many times the chosen action is repeated. For example, action = left, n = 3 -> go left 3 times....
GeorgeWTrump's user avatar
3 votes
3 answers
5k views

Why is there tanh(x)*sigmoid(x) in a LSTM cell?

CONTEXT I was wondering why there are sigmoid and tanh activation functions in an LSTM cell. My intuition was based on the flow of tanh(x)*sigmoid(x) and the ...
MASTER OF CODE's user avatar
5 votes
2 answers
476 views

Why does the activation function for a hidden layer in a MLP have to be non-polynomial?

Across multiple pieces of literature describing MLPs or while describing the universal approximation theorem, the statement is very specific on the activation function being non-polynomial. Is there a ...
niil87's user avatar
  • 53
1 vote
1 answer
3k views

Why identity function is generally treated as an activation function?

It is known that the primary purpose of activation functions, used in neural networks, is to introduce non-linearity. Then how can the linear activation function, especially the identity function, be ...
hanugm's user avatar
  • 3,880
4 votes
1 answer
1k views

Why should one ever use ReLU instead of PReLU?

To me, it seems that PReLU is strictly better than ReLU. It does not have the dying ReLU problem, it allows negative values and it has trainable parameters (which are computationally negligible to ...
algebruh's user avatar
0 votes
0 answers
346 views

Is it possible to use Softmax as an activation function for actor (policy) network in TD3 or SAC Reinforcement learning algorithms?

As I understand from literature, normally, the last activation in an actor (policy) network in TD3 and SAC algorithms is a Tanh function, which is scaled by a certain limit. My action vector is ...
Bi0max's user avatar
  • 101
2 votes
2 answers
1k views

Does the output layer in a deep neural network need an activation function?

I have enrolled in a course that uses only one hidden layer, and that is the only layer that has activation functions. The model can be visualized as follows: and here is a PyTorch implementation: <...
Omar Zayed's user avatar
2 votes
1 answer
150 views

What is meant by "well-behaved gradient" in this context?

Consider the following statement (from the paper Generative Adversarial Nets) about the success of discriminative models So far, the most striking successes in deep learning have involved ...
hanugm's user avatar
  • 3,880
2 votes
1 answer
610 views

What does "linear unit" mean in the names of activation functions?

Activation functions, in neural networks, are used to introduce non-linearity. Many activation functions that are used in neural networks have the term "Linear Unit" in their full form. &...
hanugm's user avatar
  • 3,880
0 votes
1 answer
489 views

Is my understanding on "smooth approximation" correct?

Consider the following details regarding Softplus activation function $$\text{Softplus}(x) = \dfrac{\log(1+e^{\beta x})}{\beta}$$ SoftPlus is a smooth approximation to the ReLU function and can be ...
hanugm's user avatar
  • 3,880
2 votes
0 answers
255 views

Using a rectified Tanh to train a CNN?

I have been experimenting with activation functions on CNN, and it occurred to me to use a rectified tanh function. So that is basically if z > 0 tanh(z) else 0. ...
Physics's user avatar
  • 21
1 vote
0 answers
76 views

Is the main difference between the logistic regression and the perceptron the activation function they use?

I went through a Stats StackExchange's post about the difference between logistic regression and perceptron, which is too long to get the key point. I'd like to consider the question in terms of the ...
JJJohn's user avatar
  • 221
3 votes
1 answer
1k views

Why is tanh a "smoothly" differentiable function?

The sigmoid, tanh, and ReLU are popular and useful activation functions in the literature. The following excerpt taken from p4 of Neural Networks and Neural Language Models says that ...
hanugm's user avatar
  • 3,880
1 vote
1 answer
672 views

An explanation involving the sign activation, its affect on the loss function, and the perceptron and perceptron criterion: what is this saying?

I am currently studying the textbook Neural Networks and Deep Learning by Charu C. Aggarwal. Chapter 1.2.1.3 Choice of Activation and Loss Functions says the following: The classical activation ...
The Pointer's user avatar
1 vote
0 answers
39 views

Are there guiding principles as to which activation functions suit a given RL algorithm?

Are there rules of thumb as to which activation functions work well (or which one would not) on the policy and value network of a class of RL algorithms? For hidden layers and for the output layer. ...
mugoh's user avatar
  • 531
2 votes
3 answers
462 views

Where does the so-called 'loss' / 'loss function' fit into the idea of a perceptron / artificial neuron (as presented in the figure)?

I am currently studying the textbook Neural Networks and Deep Learning by Charu C. Aggarwal. Chapter 1.2.1.3 Choice of Activation and Loss Functions presents the following figure: $\overline{X}$ is ...
The Pointer's user avatar
2 votes
1 answer
977 views

How do sigmoid functions make it so that the prediction $\hat{y}$ indicates the probability that the observed value, $y$, is $1$?

I am currently studying the textbook Neural Networks and Deep Learning by Charu C. Aggarwal. Chapter 1.2.1.3 Choice of Activation and Loss Functions says the following: The choice of activation ...
The Pointer's user avatar