17 votes
Accepted

What activation function does the human brain use?

The thing you were reading about is known as the action potential. It is a mechanism that governs how information flows within a neuron. It works like this: Neurons have an electrical potential, ...
user avatar
15 votes
Accepted

How to choose an activation function for the hidden layers?

It seems to me that you already understand the shortcomings of ReLUs and sigmoids (like dead neurons in the case of plain ReLU). You may want to look at ELU (exponential linear units) and SELU (self-...
user avatar
11 votes
Accepted

What does it mean for a neuron in a neural network to be activated?

A neuron is said activated when its output is more than a threshold, generally 0. For examples : \begin{equation} y = Relu(a) > 0 \end{equation} when \begin{equation} a = w^Tx+b > 0 \end{...
user avatar
10 votes

What is the purpose of an activation function in neural networks?

If you only had linear layers in a neural network, all the layers would essentially collapse to one linear layer, and, therefore, a "deep" neural network architecture effectively wouldn't be deep ...
user avatar
9 votes
Accepted

Do neurons of a neural network model a linear relationship?

In a neural network (NN), a neuron can act as a linear operator, but it usually acts as a non-linear one. The usual equation of a neuron $i$ in layer $l$ of an NN is $$o_i^l = \sigma(\mathbf{x}_i^l \...
user avatar
  • 33.8k
8 votes

Why is the derivative of the activation functions in neural networks important?

Consider a dataset $\mathcal{D}=\{x^{(i)},y^{(i)}:i=1,2,\ldots,N\}$ where $x^{(i)}\in\mathbb{R}^3$ and $y^{(i)}\in\mathbb{R}$ $\forall i$ The goal is to fit a function that best explains our dataset....
user avatar
  • 81
6 votes

What is the purpose of an activation function in neural networks?

Let's first talk about linearity. Linearity means the map (a function), $f: V \rightarrow W$, used is a linear map, that is, it satisfies the following two conditions $f(x + y) = f(x) + f(y), \; x, ...
user avatar
  • 749
6 votes

Why is the derivative of the activation functions in neural networks important?

If what you are asking is what is the intuition for using the derivative in backpropagation learning, instead of an in-depth mathematical explanation: Recall that the derivative tells you a function'...
user avatar
6 votes
Accepted

Smallest possible network to approximate the $sin$ function

Before anything, the function you have wrote for the network lacks the bias variables (I'm sure you used bias to get those beautiful images, otherwise your tanh ...
user avatar
  • 430
6 votes
Accepted

What does "linear unit" mean in the names of activation functions?

Have a look at these graphics showing popular linear units (image taken from Clevert et al. 2016): You can see that these functions are linear functions for $x > 0$, that's why they are called ...
user avatar
  • 186
5 votes

Why is no activation function used at the final layer of super-resolution models?

I am not into the field of super-resolution, but I think this question applies to general neural network construction. Usually, you try to solve a classification problem or a regression problem with ...
user avatar
5 votes

Why do activation functions need to be differentiable in the context of neural networks?

No, it is not necessary that an activation function is differentiable. In fact, one of the most popular activation functions, the rectifier, is non-differentiable at zero! This can create problems ...
user avatar
5 votes
Accepted

What makes multi-layer neural networks able to perform nonlinear operations?

Nonlinear relations between input and output can be achieved by using a nonlinear activation function on the value of each neuron, before it's passed on to the neurons in the next layer.
user avatar
5 votes
Accepted

Why isn't the ElliotSig activation function widely used?

I can't speak for individual researchers, but I can guess why the community as a whole hasn't adopted this activation function. ReLU is just so incredibly cheap. This benefit continues to grow as ...
user avatar
5 votes
Accepted

Why is there a sigmoid function in the hidden layer of a neural network?

Let us suppose we have a network without any functions in between. Each layer consists of a linear function. i.e layer_output = Weights.layer_input + bias ...
user avatar
5 votes

What is the mathematical definition of an activation function?

There is no strict definition of suitability of an activation function for neural networks. Instead there are a number of desirable traits, and functions that don't meet them or come close enough may ...
user avatar
  • 23.8k
5 votes
Accepted

Is a non-linear activation function needed if we perform max-pooling after the convolution layer?

Let's first recapitulate why the function that calculates the maximum between two or more numbers, $z=\operatorname{max}(x_1, x_2)$, is not a linear function. A linear function is defined as $y=f(x) =...
user avatar
  • 33.8k
4 votes

Why do activation functions need to be differentiable in the context of neural networks?

Training While "running" a neural network can be done with any activation functions, we usually want to train it - i.e., adjust its parameters so that the result would be closer to what we desire. ...
user avatar
  • 853
4 votes

What is the purpose of an activation function in neural networks?

Consider a very simple neural network, with just 2 layers, where the first has 2 neurons and the last 1 neuron, and the input size is 2. The inputs are $x_1$ and $x_1$. The weights of the first layer ...
user avatar
4 votes

What does it mean for a neuron in a neural network to be activated?

The term "activated" is mostly used when talking about activation functions which only outputs a value (except 0) when the input to the activation function is greater than a certain treshold. ...
user avatar
4 votes
Accepted

Why can't the XOR linear inseparability problem be solved with one perceptron like this?

It can be done. The activation function of a neuron does not have to be monotonic. The activation that Rahul suggested can be implemented via a continuously differentiable function, for example $ f(s)...
user avatar
4 votes
Accepted

What is a "logit probability"?

Indeed I haven't seen the term "logit probability" used in many places other than that specific paper. So, I cannot really comment on why they're using that term / where it comes from / if anyone else ...
user avatar
  • 9,369
4 votes

Why is the derivative of the activation functions in neural networks important?

The basic (and usual) algorithm used to update the weights of the artificial neural network (ANN) is an iterative, numerical and optimization algorithm, called gradient descent, which is based on and ...
user avatar
  • 33.8k
4 votes

Do neurons of a neural network model a linear relationship?

Almost never. The sum of linear functions is another linear function, so if neurons were only linear transformations there would be basically no point to having more than one neuron per layer. Instead,...
user avatar
  • 141
4 votes

In deep learning, is it possible to use discontinuous activation functions?

Even the first artificial neural network - Rosenblatt's perceptron [1] had a discontinuous activation function. That network is in introductory chapters of many textbooks about AI. For example, ...
user avatar
4 votes
Accepted

Why do ResNets avoid the vanishing gradient problem?

Before proceeding, it's important to note that ResNets, as pointed out here, were not introduced to specifically solve the VGP, but to improve learning in general. In fact, the authors of ResNet, in ...
user avatar
  • 33.8k

Only top scored, non community-wiki answers of a minimum length are eligible