# Tag Info

Accepted

• 40.9k
Accepted

### How can we compute the gradient of max pooling with overlapping regions?

When gradients in a neural network can follow multiple paths to same parameter, the different gradient values from the sources can often be added together, because the operations in the forward ...
• 32.7k

### How to calculate the gradient penalty proposed in "Improved Training of Wasserstein GANs"?

First of all, the discriminator in WGAN does not give a value in the range $[0,1]$. Compared to the traditional discriminator, it has a linear activation in the output layer. Therefore, the authors ...
• 957

### What does it mean by "zeros the networks parameters gradients" in the context of training a neural network?

In the automatic differentiation procedure after backward pass the gradient with respect to the scalar is added to the current gradient. Without calling zero_grad you will have the sum of all ...

### Why is tf.abs non-differentiable in Tensorflow?

Creating custom gradient for tf.abs may solve the problem: ...
• 1,293
Accepted

### Mathematically speaking, Is it only the product operation used in the chain rule causing the vanishing or exploding gradient?

Your understanding is totally correct. The chain rule is defined as the product of derivatives, and as you well mention, from the mathematical point of view four scenarios can happen (you can ...
• 136
Accepted

### What specifically is the gradient of the log of the probability in policy gradient methods?

I would recommend not trying to think of this in relation to supervised learning. The policy $\pi(\cdot; \theta)$ is simply a function that is parameterised by $\theta$. If we take a $\log$ of this ...
• 4,920

### How to apply backpropagation when one layer of the network is a call-only function (no gradient)?

Well, you can specify a custom gradient by either being just the identity (i.e. returning the inputs in the gradient scope) or computing the gradient by hand if you know that expression. Otherwise, ...
• 2,948
Accepted

### In multilayer perceptron neural networks, are the names "delta", "gradient" and "error" all the same thing? or not?

The terms "error", "delta" and "gradient" in neural network back-propagation are often used as shorthand, or loose explanations for the same thing. This is not strictly ...
• 32.7k
1 vote

### How to apply backpropagation when one layer of the network is a call-only function (no gradient)?

You can maybe use a similar re-parameterisation trick. Where you may approximate the gradient rather than calculating it accurately. Introduce auxiliary variables which can mimic the function an still ...
• 135
1 vote

Your backward differentiation does not seem to follow the forward computation. I prefer marking the gradient (row vector) with a letter g (in AD literature also <...
• 261
1 vote

### What does it mean by "gradient flow" in the context of neural networks?

Here is my idea of what that means: Gradient flow is an abstract term to describe properties of the gradient. The gradient is calculated by propagating the error backwards through the networks, ...
• 1,748
1 vote
Accepted

### Which is more popular/common way of representing a gradient in AI community: as a row or column vector?

The issue doesn't come up terribly often. If you are only dealing with vectors, everything is either a row or column vector. It makes no difference which it is. A more relevant issue is whether one ...
• 1,281
1 vote

### What is the rigorous and formal definition for the direction pointed by a gradient?

If $u$ is a vector, the direction pointed by the vector is defined as $\dfrac{u}{\lVert {u}\rVert}$ where $\lVert \cdot \rVert$ is the 2 norm (euclidean norm).
• 1,281
1 vote

### How many directions of gradients exist for a function in higher dimensional space?

Let's look at the definition of gradient: In vector calculus, the gradient of a scalar-valued differentiable function $f$ of several variables is the vector field (or vector-valued function) \$\nabla ...
• 5,318
1 vote

### What all does the gradient tells us other than the direction to move parameters?

Momentum was big. It allowed several steps to be evened out so that most of the motion in the weights was in the direction of the optimum. It operates against sequential measurements of the error. ...
• 373
1 vote
Accepted

### What is the high-level algorithm followed by contemporary packages for the calculation of gradient?

Does the popular packages like PyTorch, Tensorflow, Keras, etc., use this or a variant of this algorithm to find the gradients at a particular point? Yes. This is effectively what back-propagation is....
• 32.7k
1 vote

### Is there any significance for higher order gradients in artificial intelligence?

Gradient descent presumes a Taylor Series. They estimate the loss given the inputs and target, then use the difference to move the system weights to produce a less-bad loss. The learner as a ...
• 373
1 vote

### What does it mean by strong or sufficient gradient for training in this context?

The terms "insufficient gradient" or "not strong enough gradient" usually means that the magnitude of the gradient vector is too small or nearly zero that they can't drive the ...
• 258

Only top scored, non community-wiki answers of a minimum length are eligible