13
votes
Accepted
Why is the derivative of this objective function 0 if the policy is deterministic?
Here is the gradient that they are discussing in the video:
$$\nabla_{\theta} J(\theta) \approx \frac{1}{N} \sum_{i=1}^N \left( \sum_{t=1}^T \nabla_{\theta} \log \pi_{\theta} (\mathbf{a}_{i, t} \vert \...
- 10.1k
12
votes
Accepted
What is the relationship between gradient accumulation and batch size?
There isn't any explicit relation between the batch size and the gradient accumulation steps, except for the fact that gradient accumulation helps one to fit models with relatively larger batch sizes (...
- 271
4
votes
Accepted
Why is automatic differentiation still used, if today's computers can calculate symbolic derivatives quite fast?
That depends on the point-of-view. If you have some function given as a string of its mathematical formula and the output is the fully inserted string of its derivative, then the chain rule $(u\circ v)...
- 216
4
votes
Accepted
Why is tf.abs non-differentiable in Tensorflow?
By convention, the $\mathrm{ReLU}$ activation is treated as if it is differentiable at zero (e.g. in [1]). Therefore it makes sense for TensorFlow to adopt this convention for ...
- 1,000
3
votes
Why to use gradient accumulation?
This image from here nicely illustrates how gradient accumulation is performed:
Assuming infinite memory and compute we would be able to compute the gradient on the full batch, this would provide us ...
- 413
3
votes
What does it mean by "gradient flow" in the context of neural networks?
It has. Gradient flow or more generally flow is a well known concept in maths. Say we have a function $f:\mathbb R^n \longrightarrow \mathbb R^n$ and a function $\theta:[0,\infty)\longrightarrow \...
- 131
3
votes
What specifically is the gradient of the log of the probability in policy gradient methods?
Consider a function $f(x)$ where $x$ is a random variable, whose distribution depends on $\theta$. The objective is to minimize
\begin{align*}
\mathbb{E}_x[f(x)] = \int_x f(x) \pi(x, \theta) dx
\end{...
- 1,201
3
votes
Accepted
Why is it a problem if the outputs of an activation function are not zero-centered?
Yes, if the activation function of the network is not zero centered, $y = f(x^{T}w)$ is always positive or always negative. Thus, the output of a layer is always being moved to either the positive ...
- 1,094
3
votes
Why is the derivative of this objective function 0 if the policy is deterministic?
Well, I'd rather comment, but I don't have yet this privilege, so here are some comments.
First, having a deterministic policy inside the log would do create trivial terms.
Secondly, for me, in ...
- 591
3
votes
Accepted
How can the sum of squared errors have negative gradient if it's defined as the squared of the error?
If $t^i - o^i$ is negative, doesn't the power of 2 eliminate any negative result?
In the loss function, yes that is correct, and is what you want - a measurement that gets higher due to any ...
- 30.3k
2
votes
Accepted
How is the gradient of the loss function in DQN derived?
In general, if you have a composite function $h(x) = g(f(x))$, then $\frac{dh}{dx} = \frac{d g}{df} \frac{d f}{dx}$. In your case, the function to differentiate is
$$L_{i}(\theta_{i}) = \mathbb{E}_{(s,...
- 39.5k
2
votes
Accepted
How can we compute the gradient of max pooling with overlapping regions?
When gradients in a neural network can follow multiple paths to same parameter, the different gradient values from the sources can often be added together, because the operations in the forward ...
- 30.3k
2
votes
How to calculate the gradient penalty proposed in "Improved Training of Wasserstein GANs"?
First of all, the discriminator in WGAN does not give a value in the range $[0,1]$. Compared to the traditional discriminator, it has a linear activation in the output layer. Therefore, the authors ...
- 927
2
votes
What does it mean by "zeros the networks parameters gradients" in the context of training a neural network?
In the automatic differentiation procedure after backward pass
the gradient with respect to the scalar is added to the current gradient. Without calling zero_grad you will have the sum of all ...
- 2,564
2
votes
Why is tf.abs non-differentiable in Tensorflow?
Creating custom gradient for tf.abs may solve the problem:
...
- 1,283
2
votes
Accepted
Mathematically speaking, Is it only the product operation used in the chain rule causing the vanishing or exploding gradient?
Your understanding is totally correct. The chain rule is defined as the product of derivatives, and as you well mention, from the mathematical point of view four scenarios can happen (you can ...
- 136
2
votes
Accepted
What specifically is the gradient of the log of the probability in policy gradient methods?
I would recommend not trying to think of this in relation to supervised learning.
The policy $\pi(\cdot; \theta)$ is simply a function that is parameterised by $\theta$. If we take a $\log$ of this ...
- 4,665
2
votes
How to apply backpropagation when one layer of the network is a call-only function (no gradient)?
Well, you can specify a custom gradient by either being just the identity (i.e. returning the inputs in the gradient scope) or computing the gradient by hand if you know that expression.
Otherwise, ...
- 2,443
2
votes
Accepted
In multilayer perceptron neural networks, are the names "delta", "gradient" and "error" all the same thing? or not?
The terms "error", "delta" and "gradient" in neural network back-propagation are often used as shorthand, or loose explanations for the same thing. This is not strictly ...
- 30.3k
1
vote
How to apply backpropagation when one layer of the network is a call-only function (no gradient)?
You can maybe use a similar re-parameterisation trick. Where you may approximate the gradient rather than calculating it accurately.
Introduce auxiliary variables which can mimic the function an still ...
- 135
1
vote
I’m making a simple neural network from scratch and it won’t learn anything. Please help
Your backward differentiation does not seem to follow the forward computation.
I prefer marking the gradient (row vector) with a letter g (in AD literature also <...
- 216
1
vote
What does it mean by "gradient flow" in the context of neural networks?
Here is my idea of what that means: Gradient flow is an abstract term to describe properties of the gradient. The gradient is calculated by propagating the error backwards through the networks, ...
- 1,536
1
vote
Accepted
Which is more popular/common way of representing a gradient in AI community: as a row or column vector?
The issue doesn't come up terribly often. If you are only dealing with vectors, everything is either a row or column vector. It makes no difference which it is.
A more relevant issue is whether one ...
- 1,201
1
vote
What is the rigorous and formal definition for the direction pointed by a gradient?
If $u$ is a vector, the direction pointed by the vector is defined as $\dfrac{u}{\lVert {u}\rVert}$ where $\lVert \cdot \rVert$ is the 2 norm (euclidean norm).
- 1,201
1
vote
How many directions of gradients exist for a function in higher dimensional space?
Let's look at the definition of gradient:
In vector calculus, the gradient of a scalar-valued differentiable function $f$ of several variables is the vector field (or vector-valued function) $\nabla ...
- 5,218
1
vote
What all does the gradient tells us other than the direction to move parameters?
Momentum was big. It allowed several steps to be evened out so that most of the motion in the weights was in the direction of the optimum. It operates against sequential measurements of the error. ...
- 371
1
vote
Accepted
What is the high-level algorithm followed by contemporary packages for the calculation of gradient?
Does the popular packages like PyTorch, Tensorflow, Keras, etc., use this or a variant of this algorithm to find the gradients at a particular point?
Yes. This is effectively what back-propagation is....
- 30.3k
1
vote
Is there any significance for higher order gradients in artificial intelligence?
Gradient descent presumes a Taylor Series. They estimate the loss given the inputs and target, then use the difference to move the system weights to produce a less-bad loss.
The learner as a ...
- 371
1
vote
What does it mean by strong or sufficient gradient for training in this context?
The terms "insufficient gradient" or "not strong enough gradient" usually means that the magnitude of the gradient vector is too small or nearly zero that they can't drive the ...
- 238
Only top scored, non community-wiki answers of a minimum length are eligible