People who code: we want your input. Take the Survey
14

It seems to me that you already understand the shortcomings of ReLUs and sigmoids (like dead neurons in the case of plain ReLU). You may want to look at ELU (exponential linear units) and SELU (self-normalising version of ELU). Under some mild assumptions, the latter has the nice property of self-normalisation, which mitigates the problem of vanishing and ...


10

A neuron is said activated when its output is more than a threshold, generally 0. For examples : \begin{equation} y = Relu(a) > 0 \end{equation} when \begin{equation} a = w^Tx+b > 0 \end{equation} Same goes for sigmoid or other activation functions.


9

Combining ReLU, the hyper-parameterized1 leaky variant, and variant with dynamic parametrization during learning confuses two distinct things: The comparison between ReLU with the leaky variant is closely related to whether there is a need, in the particular ML case at hand, to avoid saturation — Saturation is thee loss of signal to either zero ...


9

In a neural network (NN), a neuron can act as a linear operator, but it usually acts as a non-linear one. The usual equation of a neuron $i$ in layer $l$ of an NN is $$o_i^l = \sigma(\mathbf{x}_i^l \cdot \mathbf{w}_i^l + b_i^l),$$ where $\sigma$ is a so-called activation function, which is usually a non-linearity, but it can also be the identity ...


8

If you only had linear layers in a neural network, all the layers would essentially collapse to one linear layer, and, therefore, a "deep" neural network architecture effectively wouldn't be deep anymore but just a linear classifier. $$y = f(W_1 W_2 W_3x) = f(Wx)$$ where $W$ corresponds to the matrix that represents the network weights and biases for one ...


6

Let's first talk about linearity. Linearity means the map (a function), $f: V \rightarrow W$, used is a linear map, that is, it satisfies the following two conditions $f(x + y) = f(x) + f(y), \; x, y \in V$ $f(c x) = cf(x), \; c \in \mathbb{R}$ You should be familiar with this definition if you have studied linear algebra in the past. However, it's more ...


6

If what you are asking is what is the intuition for using the derivative in backpropagation learning, instead of an in-depth mathematical explanation: Recall that the derivative tells you a function's sensitivity to change with respect to a change in its input. A high (absolute) value for the derivative at a certain point means that the function is very ...


6

Before anything, the function you have wrote for the network lacks the bias variables (I'm sure you used bias to get those beautiful images, otherwise your tanh network had to start from zero). Generally I would say it's impossible to have a good approximation of sinus with just 3 neurons, but if you want to consider one period of sinus, then you can do ...


5

Nonlinear relations between input and output can be achieved by using a nonlinear activation function on the value of each neuron, before it's passed on to the neurons in the next layer.


5

I am not into the field of super-resolution, but I think this question applies to general neural network construction. Usually, you try to solve a classification problem or a regression problem with your neural network. For classification, you try to predict probabilities that a specific output corresponds to a specific class. Therefore, every output value ...


5

I can't speak for individual researchers, but I can guess why the community as a whole hasn't adopted this activation function. ReLU is just so incredibly cheap. This benefit continues to grow as networks grow deeper. Also, they work reasonably well. As pointed out in Searching for Activation Functions, the performance improvements of the other ...


5

Consider a dataset $\mathcal{D}=\{x^{(i)},y^{(i)}:i=1,2,\ldots,N\}$ where $x^{(i)}\in\mathbb{R}^3$ and $y^{(i)}\in\mathbb{R}$ $\forall i$ The goal is to fit a function that best explains our dataset.We can fit a simple function, as we do in linear regression. But that's different about neural networks, where we fit a complex function, say: $\begin{align}h(...


5

Let us suppose we have a network without any functions in between. Each layer consists of a linear function. i.e layer_output = Weights.layer_input + bias Consider a 2 layer neural network, the outputs from layer one will be: x2 = W1*x1 + b1 Now we pass the same input to the second layer, which will be x3 = W2x*2 + b2 Also x2 = W1*x1 + b1 Substituting ...


5

There is no strict definition of suitability of an activation function for neural networks. Instead there are a number of desirable traits, and functions that don't meet them or come close enough may perform badly in general (but those functions may still work in specific cases) If you are using gradient descent as a training method, then differentiability ...


5

Let's first recapitulate why the function that calculates the maximum between two or more numbers, $z=\operatorname{max}(x_1, x_2)$, is not a linear function. A linear function is defined as $y=f(x) = ax + b$, so $y$ linearly increases with $x$. Visually, $f$ corresponds to a straight line (or hyperplane, in the case of 2 or more input variables). If $z$ ...


4

Consider a very simple neural network, with just 2 layers, where the first has 2 neurons and the last 1 neuron, and the input size is 2. The inputs are $x_1$ and $x_1$. The weights of the first layer are $w_{11}, w_{12}, w_{21}$ and $w_{22}$. We do not have activations, so the outputs of the neurons in the first layer are \begin{align} o_1 = w_{11}x_1 + w_{...


4

First Degree Linear Polynomials Non-linearity is not the correct mathematical term. Those that use it probably intend to refer to a first degree polynomial relationship between input and output, the kind of relationship that would be graphed as a straight line, a flat plane, or a higher degree surface with no curvature. To model relations more complex ...


4

No, it is not necessary that an activation function is differentiable. In fact, one of the most popular activation functions, the rectifier, is non-differentiable at zero! This can create problems with learning, as numerical gradients calculated near a non-differentiable point can be incorrect. The "kinks" section in these lecture notes discuss the issue. ...


4

It is almost mandatory to have a differentiable activation function unless, of course, you have an alternative to training the network by back-propagating the error.


4

The term "activated" is mostly used when talking about activation functions which only outputs a value (except 0) when the input to the activation function is greater than a certain treshold. Especially when discussing ReLU the term "activated" may be used. ReLU will be "activated" when it's output is greater than 0, which is also when it's input is greater ...


4

It can be done. The activation function of a neuron does not have to be monotonic. The activation that Rahul suggested can be implemented via a continuously differentiable function, for example $ f(s) = exp(-k(1-s)^2) $ which has a nice derivative $f'(s) = 2k~(1-s)f(s)$. Here, $s=w_0~x_0+w_1~x_1$. Therefore, standard gradient-based learning algorithms are ...


4

Indeed I haven't seen the term "logit probability" used in many places other than that specific paper. So, I cannot really comment on why they're using that term / where it comes from / if anyone else uses it, but I can confirm that what they mean by "logit probability" is basically the same thing that is more commonly referred to simply as "logits": they ...


4

The basic (and usual) algorithm used to update the weights of the artificial neural network (ANN) is an iterative, numerical and optimization algorithm, called gradient descent, which is based on and requires the computation of the derivative of the function you want to find the minimum of. If the function you want to find the minimum of is multivariable, ...


4

Almost never. The sum of linear functions is another linear function, so if neurons were only linear transformations there would be basically no point to having more than one neuron per layer. Instead, every neuron applies some kind of nonlinear function to its input. There are lots of different variations, but in the end the combination of the nonlinear ...


4

Even the first artificial neural network - Rosenblatt's perceptron [1] had a discontinuous activation function. That network is in introductory chapters of many textbooks about AI. For example, Michael Negnevitsky. Artificial intelligence: a guide to intelligent systems. Second Edition shows how to train such networks on pages 170-174. Error backpropagation ...


4

Before proceeding, it's important to note that ResNets, as pointed out here, were not introduced to specifically solve the VGP, but to improve learning in general. In fact, the authors of ResNet, in the original paper, noticed that neural networks without residual connections don't learn as well as ResNets, although they are using batch normalization, which, ...


4

It is definitley possible to make the links between neurons use more complex functions. Provided those functions are differentiable, backpropagation still works, and the resulting compound function might be able to learn something useful. The general name for such a thing is a computational graph and the standardised structures used in most neural networks ...


3

The outputs of a ReLU network are always "linear" and discontinuous. They can approximate curves, but it could take a lot of ReLU units. However, at the same time, their outputs will often be interpreted as a continuous, curved output. Imagine you trained a neural network that takes $x^3$ and outputs $|x^3|$ (which is similar to a parabola). This ...


Only top voted, non community-wiki answers of a minimum length are eligible