17 votes
Accepted

What activation function does the human brain use?

The thing you were reading about is known as the action potential. It is a mechanism that governs how information flows within a neuron. It works like this: Neurons have an electrical potential, ...
16 votes
Accepted

How to choose an activation function for the hidden layers?

It seems to me that you already understand the shortcomings of ReLUs and sigmoids (like dead neurons in the case of plain ReLU). You may want to look at ELU (exponential linear units) and SELU (self-...
12 votes
Accepted

What does it mean for a neuron in a neural network to be activated?

A neuron is said activated when its output is more than a threshold, generally 0. For examples : \begin{equation} y = Relu(a) > 0 \end{equation} when \begin{equation} a = w^Tx+b > 0 \end{...
11 votes

What is the purpose of an activation function in neural networks?

If you only had linear layers in a neural network, all the layers would essentially collapse to one linear layer, and, therefore, a "deep" neural network architecture effectively wouldn't be deep ...
11 votes

Are softmax outputs of classifiers true probabilities?

Excellent question. The simple answer is no. Softmax actually produces uncalibrated probabilities. That is, they do not really represent the probability of a prediction being correct. What usually ...
11 votes
Accepted

Are softmax outputs of classifiers true probabilities?

The answer is both yes, and no. Or, to put it another way, the answer depends on what exactly you mean by "represent probabilities", and there is a valid sense in which the answer is yes, ...
  • 246
10 votes
Accepted

Do neurons of a neural network model a linear relationship?

In a neural network (NN), a neuron can act as a linear operator, but it usually acts as a non-linear one. The usual equation of a neuron $i$ in layer $l$ of an NN is $$o_i^l = \sigma(\mathbf{x}_i^l \...
  • 35.6k
9 votes

Why is the derivative of the activation functions in neural networks important?

Consider a dataset $\mathcal{D}=\{x^{(i)},y^{(i)}:i=1,2,\ldots,N\}$ where $x^{(i)}\in\mathbb{R}^3$ and $y^{(i)}\in\mathbb{R}$ $\forall i$ The goal is to fit a function that best explains our dataset....
  • 111
7 votes
Accepted

Why do ResNets avoid the vanishing gradient problem?

Before proceeding, it's important to note that ResNets, as pointed out here, were not introduced to specifically solve the VGP, but to improve learning in general. In fact, the authors of ResNet, in ...
  • 35.6k
6 votes

Why is no activation function used at the final layer of super-resolution models?

I am not into the field of super-resolution, but I think this question applies to general neural network construction. Usually, you try to solve a classification problem or a regression problem with ...
6 votes

What is the purpose of an activation function in neural networks?

Let's first talk about linearity. Linearity means the map (a function), $f: V \rightarrow W$, used is a linear map, that is, it satisfies the following two conditions $f(x + y) = f(x) + f(y), \; x, ...
  • 807
6 votes
Accepted

Why isn't the ElliotSig activation function widely used?

I can't speak for individual researchers, but I can guess why the community as a whole hasn't adopted this activation function. ReLU is just so incredibly cheap. This benefit continues to grow as ...
6 votes

Why is the derivative of the activation functions in neural networks important?

If what you are asking is what is the intuition for using the derivative in backpropagation learning, instead of an in-depth mathematical explanation: Recall that the derivative tells you a function'...
6 votes
Accepted

Smallest possible network to approximate the $sin$ function

Before anything, the function you have wrote for the network lacks the bias variables (I'm sure you used bias to get those beautiful images, otherwise your tanh ...
  • 420
6 votes
Accepted

What does "linear unit" mean in the names of activation functions?

Have a look at these graphics showing popular linear units (image taken from Clevert et al. 2016): You can see that these functions are linear functions for $x > 0$, that's why they are called ...
  • 186
5 votes
Accepted

How exactly can ReLUs approximate non-linear and curved functions?

The outputs of a ReLU network are always "linear" and discontinuous. They can approximate curves, but it could take a lot of ReLU units. However, at the same time, their outputs will often ...
5 votes

Why do activation functions need to be differentiable in the context of neural networks?

No, it is not necessary that an activation function is differentiable. In fact, one of the most popular activation functions, the rectifier, is non-differentiable at zero! This can create problems ...
5 votes
Accepted

What makes multi-layer neural networks able to perform nonlinear operations?

Nonlinear relations between input and output can be achieved by using a nonlinear activation function on the value of each neuron, before it's passed on to the neurons in the next layer.
5 votes
Accepted

What is a "logit probability"?

Indeed I haven't seen the term "logit probability" used in many places other than that specific paper. So, I cannot really comment on why they're using that term / where it comes from / if anyone else ...
  • 9,604
5 votes

Do neurons of a neural network model a linear relationship?

Almost never. The sum of linear functions is another linear function, so if neurons were only linear transformations there would be basically no point to having more than one neuron per layer. Instead,...
  • 151
5 votes
Accepted

Why is there a sigmoid function in the hidden layer of a neural network?

Let us suppose we have a network without any functions in between. Each layer consists of a linear function. i.e layer_output = Weights.layer_input + bias ...
5 votes

What is the mathematical definition of an activation function?

There is no strict definition of suitability of an activation function for neural networks. Instead there are a number of desirable traits, and functions that don't meet them or come close enough may ...
5 votes
Accepted

Is a non-linear activation function needed if we perform max-pooling after the convolution layer?

Let's first recapitulate why the function that calculates the maximum between two or more numbers, $z=\operatorname{max}(x_1, x_2)$, is not a linear function. A linear function is defined as $y=f(x) =...
  • 35.6k
4 votes

What does it mean for a neuron in a neural network to be activated?

The term "activated" is mostly used when talking about activation functions which only outputs a value (except 0) when the input to the activation function is greater than a certain treshold. ...
4 votes

Why do activation functions need to be differentiable in the context of neural networks?

Training While "running" a neural network can be done with any activation functions, we usually want to train it - i.e., adjust its parameters so that the result would be closer to what we desire. ...
  • 853
4 votes

What is the purpose of an activation function in neural networks?

Consider a very simple neural network, with just 2 layers, where the first has 2 neurons and the last 1 neuron, and the input size is 2. The inputs are $x_1$ and $x_1$. The weights of the first layer ...

Only top scored, non community-wiki answers of a minimum length are eligible