Skip to main content
OverflowAI is here! AI power for your Stack Overflow for Teams knowledge community. Learn more

Questions tagged [activation-functions]

For questions related to the selection of and theory behind specific activation functions used in artificial networks.

37 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
4 votes
0 answers
258 views

Why are most commonly used activation functions continuous?

I have come to notice that the most commonly used activation functions are continuous. Is there any specific reason behind this? Results such as this paper have worked on training networks with ...
ABIM's user avatar
  • 565
4 votes
0 answers
206 views

Has the logistic map ever been used as an activation function?

I find the logistic map absolutely fascinating. Both in itself (because I love fractal) and because it is observed in nature (see: https://www.youtube.com/watch?v=ovJcsL7vyrk). I'm wondering if anyone ...
ker2x's user avatar
  • 163
3 votes
0 answers
85 views

Would a different learning rate for every neuron and layer mitigate or solve the vanishing gradient problem?

I'm interested in using the sigmoid (or tanh) activation function instead of RELU. I'm aware of RELU advantages on faster computation and no vanishing gradient problem. But about vanishing gradient, ...
Rogelio Triviño's user avatar
3 votes
0 answers
259 views

If we use a perceptron with a non-monotonic activation function, can it solve the XOR problem?

I found several papers about how to build a perceptron able to solve the XOR problem. The papers describe a solution where the heaviside step function is replaced by a non-monotonic activation ...
app_idea54's user avatar
2 votes
1 answer
150 views

What is meant by "well-behaved gradient" in this context?

Consider the following statement (from the paper Generative Adversarial Nets) about the success of discriminative models So far, the most striking successes in deep learning have involved ...
hanugm's user avatar
  • 3,880
2 votes
0 answers
256 views

Using a rectified Tanh to train a CNN?

I have been experimenting with activation functions on CNN, and it occurred to me to use a rectified tanh function. So that is basically if z > 0 tanh(z) else 0. ...
Physics's user avatar
  • 21
2 votes
0 answers
64 views

Given a 2-layer GCN, can we choose the dimensions of the 2nd weight matrix, such that this architecture has the same capacity as a 1-layer GCN?

This might be more of a question about nested function classes: For $k$ class node classification in a graph with $n$ nodes, and $d$ feature vector. I want to compare Architecture I: the GCN model of ...
Tinatim's user avatar
  • 121
2 votes
0 answers
59 views

What is the intuition behind equations 10, 11 and 12 of the paper "Noise2Noise: Learning Image Restoration without Clean Data"?

Can anyone help me understand these functions described in the paper Noise2Noise: Learning Image Restoration without Clean Data I have read the portion A.4 in the appendix but need a more detailed and ...
Markov's user avatar
  • 41
2 votes
0 answers
353 views

How to decide if gradients are vanishing?

I am trying to debug a convolutional neural network. I am seeing gradients close to zero. How can I decide whether these gradients are vanishing or not? Is there some threshold to decide on vanishing ...
pramesh's user avatar
  • 121
2 votes
1 answer
158 views

What work has been done with Poisson-style regression via neural networks with exponential activation functions?

The first neural net I wrote was a classifier. After that, I learned that neural nets can be used for regression tasks, even quantile regression. It has become clear to me that the usual games with ...
Dave's user avatar
  • 618
2 votes
0 answers
337 views

Which activation functions should I use for polynomial regression?

I am a beginner in machine learning and neural networks. I have only used neural networks for classification problems. My aim is to modify it so that it can work for polynomial regression as well. In ...
isaac john's user avatar
2 votes
0 answers
132 views

Are there any commonly used discontinuous activation functions?

Are there any commonly used activation functions (e.g. that take values in $(0,.5)\cup (.5,1)$)? Preferably for classification? Why? I was looking for commonly used activation functions on Google, ...
ABIM's user avatar
  • 565
2 votes
0 answers
33 views

Language Learning feedback with AI

Is there a program under development that uses AI technology, like Siri, to "hold hands" so to speak with a language learner and coach them on accent, colloqiual expressions, or to let them guide the ...
Tristan Beckwith's user avatar
1 vote
1 answer
36 views

How do we get from types of activation functions to fitting lines to our data?

I'm completely new to AI and admittedly have never been good at math (also please excuse me if I use the wrong terminology). Despite this, I'm trying wrap my head around activation functions and how ...
Garrett's user avatar
  • 181
1 vote
0 answers
26 views

Can an activation function with large derivative cause exploding gradient?

The maximum derivative of most of the currently existing activation functions is around 1. Can an activation function with derivatives higher than 1, say 1000 (a), cause exploding gradient problem? ...
JGM's user avatar
  • 11
1 vote
0 answers
142 views

Neural network and logical gates

I have a network witch consist of two fully connected layers (without bias) and a ReLu function in between. The network input is two binary numbers, and the output should be the a logical gate result: ...
Daniel's user avatar
  • 131
1 vote
0 answers
76 views

Is the main difference between the logistic regression and the perceptron the activation function they use?

I went through a Stats StackExchange's post about the difference between logistic regression and perceptron, which is too long to get the key point. I'd like to consider the question in terms of the ...
JJJohn's user avatar
  • 221
1 vote
0 answers
39 views

Are there guiding principles as to which activation functions suit a given RL algorithm?

Are there rules of thumb as to which activation functions work well (or which one would not) on the policy and value network of a class of RL algorithms? For hidden layers and for the output layer. ...
mugoh's user avatar
  • 531
1 vote
0 answers
59 views

Is there literature on Neural Network with activation functions of bounded domain?

I think to have found a somewhat interesting connection between neural networks and another area of mathematics. However, it requires the activation functions in the network to have a bounded - ...
olukatorzu's user avatar
1 vote
0 answers
196 views

What does Dice Loss should receive in case of binary segmentation

I implemented Dice loss class in pytorch: ...
David's user avatar
  • 188
1 vote
0 answers
135 views

What are the benefits of using ELU over other activation functions in CNNs?

I have come up with some examples of CNNs (segmentation CNNs) that use ELU (exponential linear unit) as an activation function. What are the benefits of this activation function over others, such as ...
IgnacioGaBo's user avatar
1 vote
0 answers
90 views

Dynamically adapting activation function

I am training a network through reinforcement learning. The policy network learns rotations, but depending on the actual input (state), the output of the network should be restricted to be in certain ...
thsolyt's user avatar
  • 31
1 vote
0 answers
105 views

If the output of a model is a ridge function, what should the activation functions at all the nodes be?

I have the following assignment. I can't understand the b part of this question in my assignment. I have completed the 1st part and understand the maths behind it, but the 2nd part has me stumped. I ...
hey dude's user avatar
1 vote
0 answers
174 views

Can SqueezeNet be used for regression?

I want a model that outputs the pixel coordinates of the tip of my forefinger, and whether it's touching something or not. Those would be 3 output neurons: 2 for the X-Y coordinates and 1, with a ...
Orly's user avatar
  • 71
1 vote
1 answer
339 views

Are PreLU and Leaky ReLU better than ReLU in the case of noisy labels?

Let's assume I want to build a semantic segmentation algorithm, based on Multires-UNET. My GT-masks are messy and generated by a GAN, but they are getting better and better over time. The goal is ...
Paul Higazi's user avatar
1 vote
0 answers
760 views

Which activation functions can lead to the vanishing gradient problem?

From this video tutorial Vanishing Gradient Tutorial, the sigmoid function and the hyperbolic tangent can produce the vanishing gradient problem. What other activation functions can lead to the ...
DRV's user avatar
  • 1,683
1 vote
0 answers
61 views

What are the pros and cons of the common activation functions?

I have heard that sigmoid activation functions should not be used on neural networks with many hidden layers as the gradients tend to vanish in deep networks. When should each of the common ...
KaneM's user avatar
  • 309
1 vote
0 answers
33 views

What activation functions are better for what problems?

I’ve been reading about neural network architectures. In certain cases, people say that the sigmoid "more accurately reflects real-life" and, in other cases, functions like hard limits reflect "the ...
user8714896's user avatar
1 vote
0 answers
39 views

How do I choose the activation function of the output layer of a neural network (based on theoretical motivations)?

Is there a method (not a table of recommendations!) that could tell me what activation function to choose if the outputs of the neural network have some interpretation? For example, these can be the ...
Josef Ondrej's user avatar
0 votes
0 answers
22 views

How does the relu nonlinearity allow for nonlinear polynomial transformations?

Since the neural network nonlinearities allow for nonlinear transformations that can stretch and squish the function, how can the ReLU activation function do this? I think for the sigmoid nonlinearity ...
Vivek Reddy's user avatar
0 votes
0 answers
33 views

Multi-Task VAE, Decoder Activation Functions?

I'm working on a Multi-Task VAE with one Encoder and two Decoders. The input consists of a vector with parameters which describe a design of a fluid system. The goal is to reconstruct the parameters ...
tekay's user avatar
  • 1
0 votes
1 answer
45 views

Is It Reasonable to Get NaN Outputs With Identity Activation Functions

I made my own Neural Network from scratch in unity with C# and I am using it as DQN. I set up my network which has 4 layers: 9 input values, 20 nodes in the second layer, 15 nodes in the third layer, ...
Ege's user avatar
  • 25
0 votes
0 answers
57 views

What is actually tuned during prompt engineering of autoregressive LLMs like GPT?

There are a lot of sites going over prompt engineering, but I don't see them explaining what is actually being changed. Is it hidden activation layers being tuned?
user14094230's user avatar
0 votes
0 answers
131 views

What are the advantages and disadvantages of higher order neuron activation functions?

I've been reading about different types of neurons that the traditional linear one. One example I came across is the Sigma-Pi neuron, where the activation function includes higher order terms, such as ...
burn_burn_55's user avatar
0 votes
0 answers
346 views

Is it possible to use Softmax as an activation function for actor (policy) network in TD3 or SAC Reinforcement learning algorithms?

As I understand from literature, normally, the last activation in an actor (policy) network in TD3 and SAC algorithms is a Tanh function, which is scaled by a certain limit. My action vector is ...
Bi0max's user avatar
  • 101
0 votes
1 answer
280 views

Why can a neural network use more than one activation function?

From trying to understand neural networks better, I've come upon a tentative notion that an activation function aims to build a function it's approximating via linear combinations with biases and ...
sangstar's user avatar
0 votes
0 answers
42 views

Why is sine activation function not used frequently since we know from fourier transforms that sine functions can combine to fit any function?

Pretty much the title. I'm no expert but from what I know, if you add up enough sine functions with proper amplitudes and frequencies you can get any function you want as a result. With that knowledge,...
user1477107's user avatar