# Questions tagged [activation-function]

For questions related to the selection of and theory behind specific activation functions used in artificial networks.

133 questions
Filter by
Sorted by
Tagged with
216 views

### How do two perceptrons produce different linear decision boundaries when learning?

I've learned that you can use two perceptrons to ultimately create a classifier for non-linearly separable data. I'm trying to understand how / if these two perceptrons converge to two different ...
36 views

### How do I choose the activation function of the output layer of a neural network (based on theoretical motivations)?

Is there a method (not a table of recommendations!) that could tell me what activation function to choose if the outputs of the neural network have some interpretation? For example, these can be the ...
46 views

### Is it a great misconception that the softmax is an activation function?

An activation function is a function from $R \rightarrow R$. It takes as input the inner products of weights and activations in the previous layer. It outputs the activation. A softmax however, is a ...
500 views

### Why isn't the ElliotSig activation function widely used?

The Softsign (a.k.a. ElliotSig) activation function is really simple: $$f(x) = \frac{x}{1+|x|}$$ It is bounded $[-1,1]$, has a first derivative, it is monotonic, and it is computationally ...
632 views

### What is a “logit probability”?

DeepMind's paper "Mastering the game of Go without human knowledge" states in its "Methods" section on its "Neural network architecture" that the output layer of AlphaGo Zero's policy head is "A fully ...
2k views

### What happens when I mix activation functions?

There are several activation functions, such as ReLU, sigmoid or $\tanh$. What happens when I mix activation functions? I recently found that Google has developed Swish activation function which is (...
182 views

### Method to compute the sum when the activation is a continuous function?

Background My understanding is the input neurons seem to seem to compute a weighted sum moving from one layer to another. $$\sum_i a_i w_i = a'_{k}$$ But to compute this weighted sum the sum ...
377 views

### What is the derivative function used in backpropagration?

I'm learning AI, but this confuses me. The derivative function used in backpropagation is the derivative of activation function or the derivative of loss function? These terms are confusing: ...
158 views

### If we use a perceptron with a non-monotonic activation function, can it solve the XOR problem?

I found several papers about how to build a perceptron able to solve the XOR problem. The papers describe a solution where the heaviside step function is replaced by a non-monotonic activation ...
657 views

### Why can't the XOR linear inseparability problem be solved with one perceptron like this?

Consider a perceptron where $w_0=1$ and $w_1=1$: Now, suppose that we use the following activation function \begin{align} f(x)= \begin{cases} 1, \text{ if }x =1\\ 0, \text{ otherwise} \end{cases} \...
508 views

### Which neural network should I use to approximate a specific function?

We have convolutional neural networks and recurrent neural networks for analysing respectively images and sequential data. How do I determine which neural network architecture is more appropriate to ...
868 views

### Is it suitable to find inverse of last layer's activation function and apply it on the target output?

I have a neural network with the following structure: I am expecting specific outputs from the neural network which are the target values for my training. Let's say the target values are 0.8 for the ...
573 views

### Is the cube root function suitable as a n activation function?

I am trying to design a neural network on Python. Instead of the sigmoid function which has a limited range, I am thinking of using the cube root function which has the following graph: Is this ...
1k views

### Is a linear activation function (in the output layer) equivalent to an identity function?

I have a simple question about the choice of activation function for the output layer in feed-forward neural networks. I have seen several codes where the choice of the activation function for the ...
236 views

### Should the input to the negative log likelihood loss function be probabilities?

I am trying to train a supervised model where the output from the model is output of a linear function $WX + b$. Kindly note that I'm not using any softmax or $\log$ softmax on the result of the ...
143 views

### Why do non-linear activation functions not require a specific non-linear relation between its inputs and outputs?

A linear activation function (or none at all) should only be used when the relation between input and output is linear. Why doesn't the same rule apply for other activation functions? For example, why ...
614 views

### Target values of 0.1 for 0 and 0.9 for 1 for sigmoid

I recently read an article about neural networks saying that, when using sigmoid as activation function, it's advised to use 0.1 as target value instead of 0, and 0.9 instead of 1. This was to avoid "...
86 views

### What makes multi-layer neural networks able to perform nonlinear operations?

As I know, a single layer neural network can only do linear operations, but multilayered ones can. Also, I recently learned that finite matrices/tensors, which are used in many neural networks, can ...
246 views

### What does it mean for a neuron in a neural network to be activated?

I just stumbled upon the concept of neuron coverage, which is the ratio of activated neurons and total neurons in a neural network. But what does it mean for a neuron to be "activated"? I know what ...
15k views

### What are the advantages of ReLU vs Leaky ReLU and Parametric ReLU (if any)?

I think that the advantage of using Leaky ReLU instead of ReLU is that in this way we cannot have vanishing gradient. Parametric ReLU has the same advantage with the only difference that the slope of ...
9k views

### How to choose an activation function for the hidden layers?

I choose the activation function for the output layer depending on the output that I need and the properties of the activation function that I know. For example, I choose the sigmoid function when I'm ...
276 views

### Which functions can be activation functions?

What are the required characteristics of an activation function (in a neural network)? Which functions can be activation functions? For example, which of the functions below can be used as an ...
5k views

### Why do we prefer ReLU over linear activation functions?

The ReLU activation function is defined as follows $$y = \operatorname{max}(0,x)$$ And the linear activation function is defined as follows $$y = x$$ The ReLU nonlinearity just clips the values ...
83 views

### Why don't ELUs multiply the linear portion by $\alpha$?

An exponential linear unit (as proposed by Clevert et al.) uses the function: \begin{align} \text{ELU}_\alpha(x) = \begin{cases} \alpha(e^x - 1), &\text{if } x < 0\\ x, \text{if} &\text{if ...
122 views

### ANNs with multiple activation outputs

Interested to know if there was any use or interest in activation functions with more than one output value to the next column instead of single firing. I'm interested to know if this would have any ...
2k views

### How exactly can ReLUs approximate non-linear and curved functions?

Currently, the most commonly used activation functions are ReLUs. So I answered this question What is the purpose of an activation function in neural networks? and, while writing the answer, it struck ...
17k views

### What is the purpose of an activation function in neural networks?

It is said that activation functions in neural networks help introduce non-linearity. What does this mean? What does non-linearity mean in this context? How does the introduction of this non-...
4k views

### What is the derivative of the Leaky ReLU activation function?

I am implementing a feed-forward neural network with leaky ReLU activation functions and back-propagation from scratch. Now, I need to compute the partial derivatives, but I don't know what the ...
2k views

### Why is no activation function used at the final layer of super-resolution models?

I'm trying to implement some image super-resolution models on medical images. After reading a set of papers, I found that none of the existing models use any activation function for the last layer. ...
223 views

### Why do non-linear activation functions that produce values larger than 1 or smaller than 0 work?

Why do non-linear activation functions that produce values larger than 1 or smaller than 0 work? My understanding is that neurons can only produce values between 0 and 1, and that this assumption can ...
824 views

### What are the counterparts of non-linearities and dropout in fully convolutional networks?

I am trying to replicate the fully convolutional networks (FCN) concept described here for semantic segmentation. It seems people have successfully trained such models by removing fully connected ...