Skip to main content

Questions tagged [activation-functions]

For questions related to the selection of and theory behind specific activation functions used in artificial networks.

Filter by
Sorted by
Tagged with
41 votes
5 answers
26k views

What is the purpose of an activation function in neural networks?

It is said that activation functions in neural networks help introduce non-linearity. What does this mean? What does non-linearity mean in this context? How does the introduction of this non-...
Mohsin's user avatar
  • 982
24 votes
3 answers
12k views

How to choose an activation function for the hidden layers?

I choose the activation function for the output layer depending on the output that I need and the properties of the activation function that I know. For example, I choose the sigmoid function when I'm ...
gvgramazio's user avatar
3 votes
1 answer
1k views

Why is tanh a "smoothly" differentiable function?

The sigmoid, tanh, and ReLU are popular and useful activation functions in the literature. The following excerpt taken from p4 of Neural Networks and Neural Language Models says that ...
hanugm's user avatar
  • 3,890
2 votes
3 answers
474 views

Where does the so-called 'loss' / 'loss function' fit into the idea of a perceptron / artificial neuron (as presented in the figure)?

I am currently studying the textbook Neural Networks and Deep Learning by Charu C. Aggarwal. Chapter 1.2.1.3 Choice of Activation and Loss Functions presents the following figure: $\overline{X}$ is ...
The Pointer's user avatar
13 votes
1 answer
3k views

How exactly can ReLUs approximate non-linear and curved functions?

Currently, the most commonly used activation functions are ReLUs. So I answered this question What is the purpose of an activation function in neural networks? and, while writing the answer, it struck ...
user avatar
7 votes
1 answer
12k views

Why do ResNets avoid the vanishing gradient problem?

I read that, if we use the sigmoid or hyperbolic tangent activation functions in deep neural networks, we can have some problems with the vanishing of the gradient, and this is visible by the shapes ...
FraMan's user avatar
  • 199
5 votes
1 answer
1k views

Which functions can be activation functions?

What are the required characteristics of an activation function (in a neural network)? Which functions can be activation functions? For example, which of the functions below can be used as an ...
mBabaee's user avatar
  • 51
15 votes
4 answers
10k views

Why do activation functions need to be differentiable in the context of neural networks?

Why should an activation function of a neural network be differentiable? Is it strictly necessary or is it just advantageous?
user avatar
11 votes
2 answers
8k views

Why do we prefer ReLU over linear activation functions?

The ReLU activation function is defined as follows $$y = \operatorname{max}(0,x)$$ And the linear activation function is defined as follows $$y = x$$ The ReLU nonlinearity just clips the values ...
imflash217's user avatar
8 votes
1 answer
3k views

Do all neurons in a layer have the same activation function?

I'm new to machine learning (so excuse my nomenclature), and not being a python developer, I decided to jump in at the deep (no pun intended) end writing my own framework in C++. In my current design, ...
lfgtm's user avatar
  • 230
6 votes
2 answers
2k views

In deep learning, is it possible to use discontinuous activation functions?

In deep learning, is it possible to use discontinuous activation functions (e.g. one with jump discontinuity)? (My guess: for example, ReLU is non-differentiable at a single point, but it still has a ...
Gyeonghoon Ko's user avatar
6 votes
1 answer
1k views

What is a "logit probability"?

DeepMind's paper "Mastering the game of Go without human knowledge" states in its "Methods" section on its "Neural network architecture" that the output layer of AlphaGo Zero's policy head is "A fully ...
sadakatsu's user avatar
  • 163
5 votes
1 answer
1k views

Why is a softmax used rather than dividing each activation by the sum?

Just wondering why a softmax is typically used in practice on outputs of most neural nets rather than just summing the activations and dividing each activation by the sum. I know it's roughly the same ...
user8714896's user avatar
3 votes
0 answers
262 views

If we use a perceptron with a non-monotonic activation function, can it solve the XOR problem?

I found several papers about how to build a perceptron able to solve the XOR problem. The papers describe a solution where the heaviside step function is replaced by a non-monotonic activation ...
app_idea54's user avatar
2 votes
1 answer
3k views

Why do we use the softmax instead of no activation function?

Why do we use the softmax activation function on the last layer? Suppose $i$ is the index that has the highest value (in the case when we don't use softmax at all). If we use softmax and take $i$th ...
dato nefaridze's user avatar
2 votes
0 answers
136 views

Are there any commonly used discontinuous activation functions?

Are there any commonly used activation functions (e.g. that take values in $(0,.5)\cup (.5,1)$)? Preferably for classification? Why? I was looking for commonly used activation functions on Google, ...
ABIM's user avatar
  • 565
2 votes
2 answers
281 views

What kind of functions can be used as activation functions? [duplicate]

I read that functions are used as activation functions only when they are differentiable. What about the unit step activation function? So, is there any other reason a function can be used as an ...
hina munir's user avatar
2 votes
0 answers
364 views

How to decide if gradients are vanishing?

I am trying to debug a convolutional neural network. I am seeing gradients close to zero. How can I decide whether these gradients are vanishing or not? Is there some threshold to decide on vanishing ...
pramesh's user avatar
  • 121
1 vote
1 answer
742 views

An explanation involving the sign activation, its affect on the loss function, and the perceptron and perceptron criterion: what is this saying?

I am currently studying the textbook Neural Networks and Deep Learning by Charu C. Aggarwal. Chapter 1.2.1.3 Choice of Activation and Loss Functions says the following: The classical activation ...
The Pointer's user avatar
1 vote
2 answers
298 views

Why don't we use trigonometric functions for the output neurons?

Why don't we use a trigonometric function, such as $\tan(x)$, where $x$ is an element of the interval $[0,pi/2)$, instead of the sigmoid function for the output neurons (in the case of classification)?...
AC18's user avatar
  • 13
1 vote
1 answer
3k views

Why identity function is generally treated as an activation function?

It is known that the primary purpose of activation functions, used in neural networks, is to introduce non-linearity. Then how can the linear activation function, especially the identity function, be ...
hanugm's user avatar
  • 3,890
0 votes
1 answer
1k views

Why my Fully Connected Neural Network outputs the same prediction?

I have a relatively small data set comprised of $3300$ data points where each data point is a $13$ dimensional vector where the $12$ first dimensions depict a "category" by taking the form ...
Daviiid's user avatar
  • 575
0 votes
1 answer
511 views

Is my understanding on "smooth approximation" correct?

Consider the following details regarding Softplus activation function $$\text{Softplus}(x) = \dfrac{\log(1+e^{\beta x})}{\beta}$$ SoftPlus is a smooth approximation to the ReLU function and can be ...
hanugm's user avatar
  • 3,890
0 votes
1 answer
55 views

What is it about sigmoid activations in particular that allows for the keeping and forgetting of past information from different time scales?

My understanding is that normal recurrent neural networks (RNNs) are not good at keeping past information from different time scales. Furthermore, my understanding is that Gated RNNs, such as Long ...
The Pointer's user avatar