Questions tagged [activation-functions]
For questions related to the selection of and theory behind specific activation functions used in artificial networks.
162
questions
0
votes
0
answers
26
views
Multi-Task VAE, Decoder Activation Functions?
I'm working on a Multi-Task VAE with one Encoder and two Decoders. The input consists of a vector with parameters which describe a design of a fluid system. The goal is to reconstruct the parameters ...
- 1
0
votes
1
answer
64
views
Why does my activation function cause NaNs?
I implemented the following activation function:
$$\sigma(x) = x^{1/3},$$
which returns NaNs after some epochs. I think this is due to the derivative exploding close to $0$. To fix this issue, I ...
- 101
0
votes
1
answer
31
views
Is It Reasonable to Get NaN Outputs With Identity Activation Functions
I made my own Neural Network from scratch in unity with C# and I am using it as DQN. I set up my network which has 4 layers: 9 input values, 20 nodes in the second layer, 15 nodes in the third layer, ...
- 25
1
vote
1
answer
30
views
How do we get from types of activation functions to fitting lines to our data?
I'm completely new to AI and admittedly have never been good at math (also please excuse me if I use the wrong terminology). Despite this, I'm trying wrap my head around activation functions and how ...
- 181
0
votes
2
answers
278
views
LUT-Based Sigmoid and Tanh Activation-Functions in Integer Quantized Networks
I want to understand how activation functions, specifically tanh and sigmoid, are used in int8 quantized neural networks. Even more specific, I want to understand a Look-up-Table based approach.
My ...
- 3
0
votes
0
answers
47
views
What is actually tuned during prompt engineering of autoregressive LLMs like GPT?
There are a lot of sites going over prompt engineering, but I don't see them explaining what is actually being changed. Is it hidden activation layers being tuned?
- 121
9
votes
2
answers
786
views
Why use ReLU over Leaky ReLU?
From my understanding a leaky ReLU attempts to address issues of vanishing gradients and nonzero-centeredness by keeping neurons that fire with a negative value alive.
With just this info to go off of,...
- 93
0
votes
0
answers
35
views
Difference in gradient calculation for the last layer activation in neural networks
I'm currently working on implementing a neural network using the sigmoid activation function and the binary cross-entropy cost function. In my implementation, I've noticed that the gradient ...
2
votes
2
answers
522
views
How do we determine the slope for leakyrelu activation function?
I am using LeakyReLU activation function in my architecture. We know that the default slope value is 1e-2. I want to understand ...
- 33
3
votes
2
answers
1k
views
Can LSTM model use ReLU or LeakyReLU as the activation funtion?
Can LSTM model use ReLU or LeakyReLU as the activation funtion? If so, when should one use tanh and when should one use ReLU or LeakyReLU?
- 67
1
vote
0
answers
22
views
Can an activation function with large derivative cause exploding gradient?
The maximum derivative of most of the currently existing activation functions is around 1. Can an activation function with derivatives higher than 1, say 1000 (a), cause exploding gradient problem? ...
- 11
2
votes
1
answer
328
views
Why do activation functions in neural networks have to be non-polynomial to approximate any function?
Can someone give me the main idea of the paper Multilayer Feedforward Networks With a Nonpolynomial Activation Function Can Approximate Any Function? I'm having trouble understanding it.
- 31
17
votes
2
answers
3k
views
Are softmax outputs of classifiers true probabilities?
BACKGROUND: The softmax function is the most common choice for an activation function for the last dense layer of a multiclass neural network classifier. The outputs of the softmax function have ...
- 922
3
votes
1
answer
94
views
Why aren't artificial derivatives used more often to solve the vanishing gradient problem?
While looking into the vanishing gradient problem, I came across a paper (https://ieeexplore.ieee.org/abstract/document/9336631) that used artificial derivatives in lieu of the real derivatives. For a ...
0
votes
1
answer
58
views
Why is the derivative of activation function all positive?
All the activation functions I see have positive derivatives.
Will negative ReLU work as well as its positive counterpart or will it lead to instability?
- 101
1
vote
0
answers
134
views
Neural network and logical gates
I have a network witch consist of two fully connected layers (without bias) and a ReLu function in between. The network input is two binary numbers, and the output should be the a logical gate result:
...
- 131
2
votes
1
answer
614
views
Why cannot linear activation functions be used to approximate any function?
In neural networks we use nonlinear activation functions such as sigmoid, ReLU, etc. Using a combination of these functions (with required scaling and shifting), we manage to estimate any nonlinear ...
- 35
2
votes
1
answer
181
views
Why and when do we use ReLU over tanh activation function?
I was reading LeCun Efficient Backprop and the author repeated stressed the importance of average the input patterns at 0 and thus justified the usage of tanh sigmoid. But if tanh is good then how ...
0
votes
1
answer
69
views
Where does the "rectified" in ReLU come from?
ReLU stands for Rectified Linear Unit. Linear Unit, I understand, since the function is piecewise linear. But what does rectified mean?
I looked up the definition and it said:
denoting an electric ...
- 123
1
vote
1
answer
169
views
Why are SVMs / Softmax classifiers considered linear while neural networks are non-linear?
My understanding is that neural networks are definitely not linear classifiers, as the point of functions like ReLU is to introduce non-linearity.
However, here's where my understanding starts to ...
- 151
0
votes
1
answer
676
views
Is there any way to train a regression model with negative values that is more stable?
I have a regression model where my target values contain roughly 60% negative values and 40% positive values. My model architecture includes a robert-large, 1 linear layer. I trained it after 1 epoch, ...
4
votes
1
answer
111
views
Why is it important/beneficial for an activation function to be zero-meaned?
Conventionally, (although there are plenty of better options) it is being said that as the choice of activation function for hidden layers, tanh should be prefered ...
- 41
0
votes
2
answers
213
views
Should the range of target values match the range of activation function used in the output layer?
Suppose I use a tansig activation function in the output layer of an artificial neural network giving me outputs in the range $[-1,1]$ and my model is applied to a binary classification problem, ...
1
vote
0
answers
206
views
How to define a custom layer in Pytorch [closed]
I am new to PyTorch and seeking your help regarding a problem I have. I need to add a costume layer to a NN in training phase. Please see the figure which shows a simple DNN with the custom layer. NN ...
- 11
0
votes
0
answers
128
views
What are the advantages and disadvantages of higher order neuron activation functions?
I've been reading about different types of neurons that the traditional linear one. One example I came across is the Sigma-Pi neuron, where the activation function includes higher order terms, such as
...
1
vote
1
answer
1k
views
Using "softmax" (non-linear) vs "linear" activation function in Deep Reinforcement Learning
I am following the tutorial in this video: https://youtu.be/cO5g5qLrLSo which implements deep reinforcement learning (DQN) to balance cart pole in OpenAI default environment.
The DQN model looks like ...
- 111
0
votes
1
answer
350
views
Are there any scale invariant activation functions that outputs probability distribution?
Softmax activation function is used to convert any random vector into a probability distribution. So, it is generally used as an activation function in the last layer of deep neural networks that are ...
- 3,612
-1
votes
1
answer
181
views
Are any non-injective activation functions used?
All activation functions I know of are injective, which I think makes sense.
But are there cases where non-injective activations can be useful?
- 133
1
vote
1
answer
828
views
How to output an integer/discrete number n with a single output neuron?
Say I have a game with 4 base actions [left, right, up, down] and then a value n, which determines how many times the chosen action is repeated.
For example, action = left, n = 3 -> go left 3 times....
- 37
3
votes
3
answers
4k
views
Why is there tanh(x)*sigmoid(x) in a LSTM cell?
CONTEXT
I was wondering why there are sigmoid and tanh activation functions in an LSTM cell.
My intuition was based on the flow of tanh(x)*sigmoid(x)
and the ...
- 242
5
votes
2
answers
408
views
Why does the activation function for a hidden layer in a MLP have to be non-polynomial?
Across multiple pieces of literature describing MLPs or while describing the universal approximation theorem, the statement is very specific on the activation function being non-polynomial.
Is there a ...
- 53
1
vote
1
answer
2k
views
Why identity function is generally treated as an activation function?
It is known that the primary purpose of activation functions, used in neural networks, is to introduce non-linearity.
Then how can the linear activation function, especially the identity function, be ...
- 3,612
4
votes
1
answer
1k
views
Why should one ever use ReLU instead of PReLU?
To me, it seems that PReLU is strictly better than ReLU. It does not have the dying ReLU problem, it allows negative values and it has trainable parameters (which are computationally negligible to ...
- 41
0
votes
0
answers
300
views
Is it possible to use Softmax as an activation function for actor (policy) network in TD3 or SAC Reinforcement learning algorithms?
As I understand from literature, normally, the last activation in an actor (policy) network in TD3 and SAC algorithms is a Tanh function, which is scaled by a certain limit.
My action vector is ...
- 101
2
votes
2
answers
1k
views
Does the output layer in a deep neural network need an activation function?
I have enrolled in a course that uses only one hidden layer, and that is the only layer that has activation functions. The model can be visualized as follows:
and here is a PyTorch implementation:
<...
- 43
1
vote
1
answer
128
views
What is meant by "well-behaved gradient" in this context?
Consider the following statement (from the paper Generative Adversarial Nets) about the success of discriminative models
So far, the most striking successes in deep learning have involved ...
- 3,612
2
votes
1
answer
538
views
What does "linear unit" mean in the names of activation functions?
Activation functions, in neural networks, are used to introduce non-linearity. Many activation functions that are used in neural networks have the term "Linear Unit" in their full form. &...
- 3,612
0
votes
1
answer
425
views
Is my understanding on "smooth approximation" correct?
Consider the following details regarding Softplus activation function
$$\text{Softplus}(x) = \dfrac{\log(1+e^{\beta x})}{\beta}$$
SoftPlus is a smooth approximation to the ReLU function and can be
...
- 3,612
2
votes
0
answers
235
views
Using a rectified Tanh to train a CNN?
I have been experimenting with activation functions on CNN, and it occurred to me to use a rectified tanh function. So that is basically if z > 0 tanh(z) else 0. ...
- 21
1
vote
0
answers
73
views
Is the main difference between the logistic regression and the perceptron the activation function they use?
I went through a Stats StackExchange's post about the difference between logistic regression and perceptron, which is too long to get the key point.
I'd like to consider the question in terms of the ...
- 221
3
votes
1
answer
933
views
Why is tanh a "smoothly" differentiable function?
The sigmoid, tanh, and ReLU are popular and useful activation functions in the literature.
The following excerpt taken from p4 of Neural Networks and Neural Language Models says that ...
- 3,612
1
vote
1
answer
333
views
An explanation involving the sign activation, its affect on the loss function, and the perceptron and perceptron criterion: what is this saying?
I am currently studying the textbook Neural Networks and Deep Learning by Charu C. Aggarwal. Chapter 1.2.1.3 Choice of Activation and Loss Functions says the following:
The classical activation ...
- 547
1
vote
0
answers
38
views
Are there guiding principles as to which activation functions suit a given RL algorithm?
Are there rules of thumb as to which activation functions work well (or which one would not) on the policy and value network of a class of RL algorithms? For hidden layers and for the output layer.
...
- 531
2
votes
3
answers
399
views
Where does the so-called 'loss' / 'loss function' fit into the idea of a perceptron / artificial neuron (as presented in the figure)?
I am currently studying the textbook Neural Networks and Deep Learning by Charu C. Aggarwal. Chapter 1.2.1.3 Choice of Activation and Loss Functions presents the following figure:
$\overline{X}$ is ...
- 547
2
votes
1
answer
857
views
How do sigmoid functions make it so that the prediction $\hat{y}$ indicates the probability that the observed value, $y$, is $1$?
I am currently studying the textbook Neural Networks and Deep Learning by Charu C. Aggarwal. Chapter 1.2.1.3 Choice of Activation and Loss Functions says the following:
The choice of activation ...
- 547
0
votes
1
answer
54
views
What is it about sigmoid activations in particular that allows for the keeping and forgetting of past information from different time scales?
My understanding is that normal recurrent neural networks (RNNs) are not good at keeping past information from different time scales. Furthermore, my understanding is that Gated RNNs, such as Long ...
- 547
0
votes
1
answer
162
views
Which solutions are there to the problem of having too large activations before the softmax (or sigmoid) layer?
I'm trying to build a neural network (NN) for classification using only N-bit integers for both the activations and weights, then I will train it with some heuristic algorithm, based only on the NN ...
- 1
1
vote
0
answers
56
views
Is there literature on Neural Network with activation functions of bounded domain?
I think to have found a somewhat interesting connection between neural networks and another area of mathematics. However, it requires the activation functions in the network to have a bounded - ...
- 11
2
votes
1
answer
2k
views
Why do we use the softmax instead of no activation function?
Why do we use the softmax activation function on the last layer?
Suppose $i$ is the index that has the highest value (in the case when we don't use softmax at all). If we use softmax and take $i$th ...
- 862
0
votes
1
answer
105
views
About the choice of the activation functions in the Multilayer Perceptron, and on what does this depends?
I've read in this: F. Rosenblatt, Principles of neurodynamics. perceptrons and the theory of brain mechanisms that in the Multilayer Perceptron the activation functions in the second, third, ..., are ...
