Skip to main content
13 votes
Accepted

Why is the derivative of this objective function 0 if the policy is deterministic?

Here is the gradient that they are discussing in the video: $$\nabla_{\theta} J(\theta) \approx \frac{1}{N} \sum_{i=1}^N \left( \sum_{t=1}^T \nabla_{\theta} \log \pi_{\theta} (\mathbf{a}_{i, t} \vert \...
Dennis Soemers's user avatar
  • 10.3k
12 votes
Accepted

What is the relationship between gradient accumulation and batch size?

There isn't any explicit relation between the batch size and the gradient accumulation steps, except for the fact that gradient accumulation helps one to fit models with relatively larger batch sizes (...
nagaK's user avatar
  • 271
4 votes
Accepted

Why is tf.abs non-differentiable in Tensorflow?

By convention, the $\mathrm{ReLU}$ activation is treated as if it is differentiable at zero (e.g. in [1]). Therefore it makes sense for TensorFlow to adopt this convention for ...
htl's user avatar
  • 1,010
4 votes

Why to use gradient accumulation?

This image from here nicely illustrates how gradient accumulation is performed: Assuming infinite memory and compute we would be able to compute the gradient on the full batch, this would provide us ...
Mariusmarten's user avatar
4 votes
Accepted

Why is automatic differentiation still used, if today's computers can calculate symbolic derivatives quite fast?

That depends on the point-of-view. If you have some function given as a string of its mathematical formula and the output is the fully inserted string of its derivative, then the chain rule $(u\circ v)...
Lutz Lehmann's user avatar
3 votes

What does it mean by "gradient flow" in the context of neural networks?

It has. Gradient flow or more generally flow is a well known concept in maths. Say we have a function $f:\mathbb R^n \longrightarrow \mathbb R^n$ and a function $\theta:[0,\infty)\longrightarrow \...
crush3dice's user avatar
3 votes

What specifically is the gradient of the log of the probability in policy gradient methods?

Consider a function $f(x)$ where $x$ is a random variable, whose distribution depends on $\theta$. The objective is to minimize \begin{align*} \mathbb{E}_x[f(x)] = \int_x f(x) \pi(x, \theta) dx \end{...
Taw's user avatar
  • 1,281
3 votes
Accepted

Why is it a problem if the outputs of an activation function are not zero-centered?

Yes, if the activation function of the network is not zero centered, $y = f(x^{T}w)$ is always positive or always negative. Thus, the output of a layer is always being moved to either the positive ...
Faizy's user avatar
  • 1,114
3 votes

Why is the derivative of this objective function 0 if the policy is deterministic?

Well, I'd rather comment, but I don't have yet this privilege, so here are some comments. First, having a deterministic policy inside the log would do create trivial terms. Secondly, for me, in ...
16Aghnar's user avatar
  • 601
3 votes
Accepted

How can the sum of squared errors have negative gradient if it's defined as the squared of the error?

If $t^i - o^i$ is negative, doesn't the power of 2 eliminate any negative result? In the loss function, yes that is correct, and is what you want - a measurement that gets higher due to any ...
Neil Slater's user avatar
  • 32.7k
2 votes
Accepted

How is the gradient of the loss function in DQN derived?

In general, if you have a composite function $h(x) = g(f(x))$, then $\frac{dh}{dx} = \frac{d g}{df} \frac{d f}{dx}$. In your case, the function to differentiate is $$L_{i}(\theta_{i}) = \mathbb{E}_{(s,...
nbro's user avatar
  • 40.9k
2 votes
Accepted

How can we compute the gradient of max pooling with overlapping regions?

When gradients in a neural network can follow multiple paths to same parameter, the different gradient values from the sources can often be added together, because the operations in the forward ...
Neil Slater's user avatar
  • 32.7k
2 votes

How to calculate the gradient penalty proposed in "Improved Training of Wasserstein GANs"?

First of all, the discriminator in WGAN does not give a value in the range $[0,1]$. Compared to the traditional discriminator, it has a linear activation in the output layer. Therefore, the authors ...
Aray Karjauv's user avatar
2 votes

What does it mean by "zeros the networks parameters gradients" in the context of training a neural network?

In the automatic differentiation procedure after backward pass the gradient with respect to the scalar is added to the current gradient. Without calling zero_grad you will have the sum of all ...
spiridon_the_sun_rotator's user avatar
2 votes

Why is tf.abs non-differentiable in Tensorflow?

Creating custom gradient for tf.abs may solve the problem: ...
Dan D.'s user avatar
  • 1,293
2 votes
Accepted

Mathematically speaking, Is it only the product operation used in the chain rule causing the vanishing or exploding gradient?

Your understanding is totally correct. The chain rule is defined as the product of derivatives, and as you well mention, from the mathematical point of view four scenarios can happen (you can ...
Jzbach's user avatar
  • 136
2 votes
Accepted

What specifically is the gradient of the log of the probability in policy gradient methods?

I would recommend not trying to think of this in relation to supervised learning. The policy $\pi(\cdot; \theta)$ is simply a function that is parameterised by $\theta$. If we take a $\log$ of this ...
David's user avatar
  • 4,920
2 votes

How to apply backpropagation when one layer of the network is a call-only function (no gradient)?

Well, you can specify a custom gradient by either being just the identity (i.e. returning the inputs in the gradient scope) or computing the gradient by hand if you know that expression. Otherwise, ...
Luca Anzalone's user avatar
2 votes
Accepted

In multilayer perceptron neural networks, are the names "delta", "gradient" and "error" all the same thing? or not?

The terms "error", "delta" and "gradient" in neural network back-propagation are often used as shorthand, or loose explanations for the same thing. This is not strictly ...
Neil Slater's user avatar
  • 32.7k
1 vote

How to apply backpropagation when one layer of the network is a call-only function (no gradient)?

You can maybe use a similar re-parameterisation trick. Where you may approximate the gradient rather than calculating it accurately. Introduce auxiliary variables which can mimic the function an still ...
Formal_this's user avatar
1 vote

I’m making a simple neural network from scratch and it won’t learn anything. Please help

Your backward differentiation does not seem to follow the forward computation. I prefer marking the gradient (row vector) with a letter g (in AD literature also <...
Lutz Lehmann's user avatar
1 vote

What does it mean by "gradient flow" in the context of neural networks?

Here is my idea of what that means: Gradient flow is an abstract term to describe properties of the gradient. The gradient is calculated by propagating the error backwards through the networks, ...
Chillston's user avatar
  • 1,748
1 vote
Accepted

Which is more popular/common way of representing a gradient in AI community: as a row or column vector?

The issue doesn't come up terribly often. If you are only dealing with vectors, everything is either a row or column vector. It makes no difference which it is. A more relevant issue is whether one ...
Taw's user avatar
  • 1,281
1 vote

What is the rigorous and formal definition for the direction pointed by a gradient?

If $u$ is a vector, the direction pointed by the vector is defined as $\dfrac{u}{\lVert {u}\rVert}$ where $\lVert \cdot \rVert$ is the 2 norm (euclidean norm).
Taw's user avatar
  • 1,281
1 vote

How many directions of gradients exist for a function in higher dimensional space?

Let's look at the definition of gradient: In vector calculus, the gradient of a scalar-valued differentiable function $f$ of several variables is the vector field (or vector-valued function) $\nabla ...
Edoardo Guerriero's user avatar
1 vote

What all does the gradient tells us other than the direction to move parameters?

Momentum was big. It allowed several steps to be evened out so that most of the motion in the weights was in the direction of the optimum. It operates against sequential measurements of the error. ...
EngrStudent's user avatar
1 vote
Accepted

What is the high-level algorithm followed by contemporary packages for the calculation of gradient?

Does the popular packages like PyTorch, Tensorflow, Keras, etc., use this or a variant of this algorithm to find the gradients at a particular point? Yes. This is effectively what back-propagation is....
Neil Slater's user avatar
  • 32.7k
1 vote

Is there any significance for higher order gradients in artificial intelligence?

Gradient descent presumes a Taylor Series. They estimate the loss given the inputs and target, then use the difference to move the system weights to produce a less-bad loss. The learner as a ...
EngrStudent's user avatar
1 vote

What does it mean by strong or sufficient gradient for training in this context?

The terms "insufficient gradient" or "not strong enough gradient" usually means that the magnitude of the gradient vector is too small or nearly zero that they can't drive the ...
ayandas's user avatar
  • 258

Only top scored, non community-wiki answers of a minimum length are eligible