Hot answers tagged

10

Combining ReLU, the hyper-parameterized1 leaky variant, and variant with dynamic parametrization during learning confuses two distinct things: The comparison between ReLU with the leaky variant is closely related to whether there is a need, in the particular ML case at hand, to avoid saturation — Saturation is thee loss of signal to either zero ...


4

Even the first artificial neural network - Rosenblatt's perceptron [1] had a discontinuous activation function. That network is in introductory chapters of many textbooks about AI. For example, Michael Negnevitsky. Artificial intelligence: a guide to intelligent systems. Second Edition shows how to train such networks on pages 170-174. Error backpropagation ...


3

The fact that features are always positive values don't guarantee that outputs of hidden layers are positive too. Due to multiplication, output of an hidden layer could contain negative values, i.e., a hidden layer can contain weights that have opposites signs as its input. Remember that only layer outputs, not their weights, are passed through ReLu, so, ...


3

Backprop through ReLU is easier than backprop through sigmoid activations. For positive activations, you just pass through the input gradients as they were. For negative activations you just set the gradients to 0. Regarding softmax, the easiest approach is to consider it a part of the negative log-likelihood loss. In other words, I am suggesting to ...


3

By convention, the $\mathrm{ReLU}$ activation is treated as if it is differentiable at zero (e.g. in [1]). Therefore it makes sense for TensorFlow to adopt this convention for tf.nn.relu. As you've found, of course, it's not true in general that we treat the gradient of the absolute value function as zero in the same situation; it makes sense for it to be an ...


3

The outputs of a ReLU network are always "linear" and discontinuous. They can approximate curves, but it could take a lot of ReLU units. However, at the same time, their outputs will often be interpreted as a continuous, curved output. Imagine you trained a neural network that takes $x^3$ and outputs $|x^3|$ (which is similar to a parabola). This ...


2

Derivative gives the rate of change in $y$ for a small change in $x$ or the slope of a function at point $x$. In the above function, y = x for x >= 0, i.e. y/x = 1 y = x/20 for x < 0, i.e. y/x = 1/20 The following function returns the derivative of leaky ReLU as explained private double leaky_relu_derivative(double x) { if (x &...


2

Why wouldn't they work? Each neuron's output is equal to a function over the sum of all its weights multiplied by their corresponding neurons. If that function is the Sigmoid function, then the output is squashed from $[0,1]$. If the entire layer uses a SoftMax function, then the output of all neurons is squashed from $[0,1]$ and their sum equals 1. In other ...


2

There are a variety of possible things that could be wrong, but let me give you some potentially useful information. Neural networks with ReLU activation functions are Turing complete for a computation with on order as many steps as the network contains nodes - for a recurrent network (an RNN), that means the same level of turing completeness as any finite ...


2

ReLU is non-linear by definition In calculus and related areas, a linear function is a function whose graph is a straight line, that is a polynomial function of degree one or zero. Since the graph of the ReLU function $f(x) = \max(0,x)$ is not a straight line (equivalently, it cannot be expressed in the form $f(x) = mx + c$), by definition it is not ...


2

Short Answer: Yes Visually: if you see the image from wikipedia, it shown that ReLU (the blue line) is non-Linear (the line is not straight, it turns in 0). You can also check "visual" definition of linear function in wikipedia: "In calculus and related areas, a linear function is a function whose graph is a straight line" Mathematically: Linear ...


2

ReLU is piecewise linear function that outputs the received input directly if it's positive, or outputs a zero. i.e., $max(0, x)$ How significant is adding relu to full connected layers? ReLU, being an activation function, will determine what the output of the nodes in your FCs are. Since it's a non-linear function, one significance is it will allow the ...


2

Yes, if there's no activation function in the last layer, the weights could simply be negative there, so the network would multiply a positive value with a negative weight, therefore outputting a negative value. There is still an activation function, but it is the identity.


2

The problem with certain activation functions, such as the sigmoid, is that they squash the input to a finite interval (i.e. they are sometimes classified as saturating activation functions). For example, the sigmoid function has codomain $[0, 1]$, as you can see from the illustration below. This property/behaviour can lead to the vanishing gradient problem ...


2

The general answer to the behavior of combining common activation functions is that the laws of calculus must be applied, specifically differential calculus, the results must be obtained through experiment to be sure of the qualities of the assembled function, and the additional complexity is likely to increase computation time. The exception to such ...


2

The most effective way to prevent both the forward and backward propagation of exploding is keeping the weights in a small range. The main way this is accomplished is through their initialization. For example in the case of He initialization, the authors show (given some assumptions) that the variance of the output of the final layer $L$ of the network is: ...


2

$F_l$ is the activation of the filter. They state in the paper that they base their method on VGG-Network, which uses ReLU as its activation function. In fact, VGG uses it in all of its hidden layers. ReLU is defined as $$f(x) = max(0,x)$$ Since ReLU is 0 for all x's below 0, the equation above holds; When x is non-positive, all terms in the loss function ...


2

I suppose, the situation is as follows - PReLU increases the expressiveness of a model for a bit at a small cost, but the gain is almost negligible as well (according to this post). There is, indeed, a noticeable difference between ReLU and PReLU, since the former takes the same value for all $\mathbb{R}_{\leq 0}$. However, compared with a LeakyReLU, note ...


1

I'm sure the biases are initially initialized to zero but I don't know how the weights are handled. Looking at the Dense layer docs: by default Dense layers biases are initialized with zeros (bias_initializer='zeros') and weights are initialized with Glorot uniform (kernel_initializer='glorot_uniform'). ... "unusual" element to point here; I've ...


1

Creating custom gradient for tf.abs may solve the problem: @tf.custom_gradient def abs_with_grad(x): y = tf.abs(x); def grad(div): # Derivation intermediate value g = 1; # Use 1 to make the chain rule just skip abs return div*g; return y,grad;


1

There is no benefit to using ReLU as the output activation of a neural network. As you said, the network will ignore training labels below zero and it will train on labels above zero as if no output activation were present. However, the problem you're describing can also occur for individual units of hidden layers, where ReLU activations are common. This is ...


1

It seems like you're suffering from the the dying ReLU problem. ReLU enforces positive values so the weights and biases your network learned are leading to a negative value passed through the ReLU function - meaning you would get 0. There are a few things you can do. I do not know the exact format of your data, but if it is MNIST it is possible you simply ...


1

You are misunderstanding something. You are mixing up inner layers with the output layer. But the question was very good. Fist of all, with the only one layer and one neuron neural networks it does not exist. Only one layer can not bring nonlinearity in the network. One neuron network means it's a linear regression or logistic regression if it passes ...


1

ReLU and sigmoid have different properties (i.e. range), as you already noticed. I've never seen the ReLU being used as the activation function of the output layer (but some people may use it for some reason, e.g. regression tasks where the output needs to be positive). ReLU is usually used as the activation function of a hidden layer. However, in your case, ...


1

I would say that it is possible, but probably not a very good idea.?Like you say, the hard requirement is that the network (and thus its components, including the activation functions) must be differentiable. ReLU isn't, but you can cheat by defining f'(0) to be 0 (or 1). A continuous function means that gradient descent leads to some local minimum¹, for ...


1

I don't think it is dead ReLU units as a main cause, although they may be happening as part of the NN failing. The NN architecture is too complex for the given task (too deep, too many neurons) and that means that any problems you have with other design choices will tend to get amplified. It could be that your NN is close to diverging on the given data and ...


1

Lets assume we have 3 Dense layers, where the activations are $x^0 \rightarrow x^1 \rightarrow x^2$, such that $x^2 = \psi PReLU(x^1) + \gamma$ and $x^1 = PReLU(Ax^0 + b)$ Now lets see what it would take to conform the PReLU into a ReLU $\begin{align*} PReLU(x^1) &= ReLU(x^1) + ReLU(p \odot x^1)\\ &= ReLU(Ax^0+b) + ReLU(p\odot(Ax^0+b))\\ &...


1

A multi-layer network in which all units have linear activation functions can always be collapsed to an equivalent network with two layers of units. That is why it is essential to use nonlinear unit activation functions. The underlying reason for using nonlinear activation functions involves a remarkable theorem of Cybenko (1989), which states that one ...


Only top voted, non community-wiki answers of a minimum length are eligible