12
votes
Accepted
Why is the derivative of this objective function 0 if the policy is deterministic?
Here is the gradient that they are discussing in the video:
$$\nabla_{\theta} J(\theta) \approx \frac{1}{N} \sum_{i=1}^N \left( \sum_{t=1}^T \nabla_{\theta} \log \pi_{\theta} (\mathbf{a}_{i, t} \vert \...
- 9,316
7
votes
Accepted
What is the relationship between gradient accumulation and batch size?
There isn't any explicit relation between the batch size and the gradient accumulation steps, except for the fact that gradient accumulation helps one to fit models with relatively larger batch sizes (...
- 211
3
votes
Accepted
Why is it a problem if the outputs of an activation function are not zero-centered?
Yes, if the activation function of the network is not zero centered, $y = f(x^{T}w)$ is always positive or always negative. Thus, the output of a layer is always being moved to either the positive ...
- 619
3
votes
Accepted
Why is tf.abs non-differentiable in Tensorflow?
By convention, the $\mathrm{ReLU}$ activation is treated as if it is differentiable at zero (e.g. in [1]). Therefore it makes sense for TensorFlow to adopt this convention for ...
- 931
3
votes
Accepted
How can the sum of squared errors have negative gradient if it's defined as the squared of the error?
If $t^i - o^i$ is negative, doesn't the power of 2 eliminate any negative result?
In the loss function, yes that is correct, and is what you want - a measurement that gets higher due to any ...
- 23.1k
3
votes
Why is the derivative of this objective function 0 if the policy is deterministic?
Well, I'd rather comment, but I don't have yet this privilege, so here are some comments.
First, having a deterministic policy inside the log would do create trivial terms.
Secondly, for me, in ...
- 571
2
votes
Accepted
How can we compute the gradient of max pooling with overlapping regions?
When gradients in a neural network can follow multiple paths to same parameter, the different gradient values from the sources can often be added together, because the operations in the forward ...
- 23.1k
2
votes
How to calculate the gradient penalty proposed in "Improved Training of Wasserstein GANs"?
First of all, the discriminator in WGAN does not give a value in the range $[0,1]$. Compared to the traditional discriminator, it has a linear activation in the output layer. Therefore, the authors ...
- 732
2
votes
What does it mean by "zeros the networks parameters gradients" in the context of training a neural network?
In the automatic differentiation procedure after backward pass
the gradient with respect to the scalar is added to the current gradient. Without calling zero_grad you will have the sum of all ...
- 1,754
2
votes
Accepted
Mathematically speaking, Is it only the product operation used in the chain rule causing the vanishing or exploding gradient?
Your understanding is totally correct. The chain rule is defined as the product of derivatives, and as you well mention, from the mathematical point of view four scenarios can happen (you can ...
- 136
2
votes
What specifically is the gradient of the log of the probability in policy gradient methods?
Consider a function $f(x)$ where $x$ is a random variable, whose distribution depends on $\theta$. The objective is to minimize
\begin{align*}
\mathbb{E}_x[f(x)] = \int_x f(x) \pi(x, \theta) dx
\end{...
- 746
1
vote
What does it mean by "gradient flow" in the context of neural networks?
Here is my idea of what that means: Gradient flow is an abstract term to describe properties of the gradient. The gradient is calculated by propagating the error backwards through the networks, ...
- 166
1
vote
Accepted
What specifically is the gradient of the log of the probability in policy gradient methods?
I would recommend not trying to think of this in relation to supervised learning.
The policy $\pi(\cdot; \theta)$ is simply a function that is parameterised by theta. If we take a $\log$ of this ...
- 4,016
1
vote
Accepted
Which is more popular/common way of representing a gradient in AI community: as a row or column vector?
The issue doesn't come up terribly often. If you are only dealing with vectors, everything is either a row or column vector. It makes no difference which it is.
A more relevant issue is whether one ...
- 746
1
vote
How many directions of gradients exist for a function in higher dimensional space?
Let's look at the definition of gradient:
In vector calculus, the gradient of a scalar-valued differentiable function $f$ of several variables is the vector field (or vector-valued function) $\nabla ...
- 3,433
1
vote
Accepted
What is the high-level algorithm followed by contemporary packages for the calculation of gradient?
Does the popular packages like PyTorch, Tensorflow, Keras, etc., use this or a variant of this algorithm to find the gradients at a particular point?
Yes. This is effectively what back-propagation is....
- 23.1k
1
vote
What does it mean by strong or sufficient gradient for training in this context?
The terms "insufficient gradient" or "not strong enough gradient" usually means that the magnitude of the gradient vector is too small or nearly zero that they can't drive the ...
- 238
1
vote
Why is tf.abs non-differentiable in Tensorflow?
Creating custom gradient for tf.abs may solve the problem:
...
- 1,133
1
vote
Is the gradient at a layer independent of the activations of the previous layers?
Is the gradient at a layer (of a feed-forward neural network) independent of the activations of the previous layers?
Yes, as per @recessive answer they are indeed independent of the previous layers.
...
- 211
1
vote
Is the gradient at a layer independent of the activations of the previous layers?
Yes, this is the premise of back-propagation, the gradient at layer $j_{n}$ is not impacted by the gradient at layer $j_{n-1}$. This allows you to start with a gradient at the output layer and ...
- 1,324
1
vote
Accepted
How is the gradient of the loss function in DQN derived?
In general, if you have a composite function $h(x) = g(f(x))$, then $\frac{dh}{dx} = \frac{d g}{df} \frac{d f}{dx}$. In your case, the function to differentiate is
$$L_{i}(\theta_{i}) = \mathbb{E}_{(s,...
- 33k
Only top scored, non community-wiki answers of a minimum length are eligible