12 votes
Accepted

Why is the derivative of this objective function 0 if the policy is deterministic?

Here is the gradient that they are discussing in the video: $$\nabla_{\theta} J(\theta) \approx \frac{1}{N} \sum_{i=1}^N \left( \sum_{t=1}^T \nabla_{\theta} \log \pi_{\theta} (\mathbf{a}_{i, t} \vert \...
user avatar
  • 9,316
7 votes
Accepted

What is the relationship between gradient accumulation and batch size?

There isn't any explicit relation between the batch size and the gradient accumulation steps, except for the fact that gradient accumulation helps one to fit models with relatively larger batch sizes (...
user avatar
  • 211
3 votes
Accepted

Why is it a problem if the outputs of an activation function are not zero-centered?

Yes, if the activation function of the network is not zero centered, $y = f(x^{T}w)$ is always positive or always negative. Thus, the output of a layer is always being moved to either the positive ...
user avatar
  • 619
3 votes
Accepted

Why is tf.abs non-differentiable in Tensorflow?

By convention, the $\mathrm{ReLU}$ activation is treated as if it is differentiable at zero (e.g. in [1]). Therefore it makes sense for TensorFlow to adopt this convention for ...
user avatar
  • 931
3 votes
Accepted

How can the sum of squared errors have negative gradient if it's defined as the squared of the error?

If $t^i - o^i$ is negative, doesn't the power of 2 eliminate any negative result? In the loss function, yes that is correct, and is what you want - a measurement that gets higher due to any ...
user avatar
  • 23.1k
3 votes

Why is the derivative of this objective function 0 if the policy is deterministic?

Well, I'd rather comment, but I don't have yet this privilege, so here are some comments. First, having a deterministic policy inside the log would do create trivial terms. Secondly, for me, in ...
user avatar
  • 571
2 votes
Accepted

How can we compute the gradient of max pooling with overlapping regions?

When gradients in a neural network can follow multiple paths to same parameter, the different gradient values from the sources can often be added together, because the operations in the forward ...
user avatar
  • 23.1k
2 votes

How to calculate the gradient penalty proposed in "Improved Training of Wasserstein GANs"?

First of all, the discriminator in WGAN does not give a value in the range $[0,1]$. Compared to the traditional discriminator, it has a linear activation in the output layer. Therefore, the authors ...
user avatar
2 votes

What does it mean by "zeros the networks parameters gradients" in the context of training a neural network?

In the automatic differentiation procedure after backward pass the gradient with respect to the scalar is added to the current gradient. Without calling zero_grad you will have the sum of all ...
user avatar
2 votes
Accepted

Mathematically speaking, Is it only the product operation used in the chain rule causing the vanishing or exploding gradient?

Your understanding is totally correct. The chain rule is defined as the product of derivatives, and as you well mention, from the mathematical point of view four scenarios can happen (you can ...
user avatar
  • 136
2 votes

What specifically is the gradient of the log of the probability in policy gradient methods?

Consider a function $f(x)$ where $x$ is a random variable, whose distribution depends on $\theta$. The objective is to minimize \begin{align*} \mathbb{E}_x[f(x)] = \int_x f(x) \pi(x, \theta) dx \end{...
user avatar
  • 746
1 vote

What does it mean by "gradient flow" in the context of neural networks?

Here is my idea of what that means: Gradient flow is an abstract term to describe properties of the gradient. The gradient is calculated by propagating the error backwards through the networks, ...
user avatar
  • 166
1 vote
Accepted

What specifically is the gradient of the log of the probability in policy gradient methods?

I would recommend not trying to think of this in relation to supervised learning. The policy $\pi(\cdot; \theta)$ is simply a function that is parameterised by theta. If we take a $\log$ of this ...
user avatar
1 vote
Accepted

Which is more popular/common way of representing a gradient in AI community: as a row or column vector?

The issue doesn't come up terribly often. If you are only dealing with vectors, everything is either a row or column vector. It makes no difference which it is. A more relevant issue is whether one ...
user avatar
  • 746
1 vote

How many directions of gradients exist for a function in higher dimensional space?

Let's look at the definition of gradient: In vector calculus, the gradient of a scalar-valued differentiable function $f$ of several variables is the vector field (or vector-valued function) $\nabla ...
user avatar
1 vote
Accepted

What is the high-level algorithm followed by contemporary packages for the calculation of gradient?

Does the popular packages like PyTorch, Tensorflow, Keras, etc., use this or a variant of this algorithm to find the gradients at a particular point? Yes. This is effectively what back-propagation is....
user avatar
  • 23.1k
1 vote

What does it mean by strong or sufficient gradient for training in this context?

The terms "insufficient gradient" or "not strong enough gradient" usually means that the magnitude of the gradient vector is too small or nearly zero that they can't drive the ...
user avatar
  • 238
1 vote

Why is tf.abs non-differentiable in Tensorflow?

Creating custom gradient for tf.abs may solve the problem: ...
user avatar
  • 1,133
1 vote

Is the gradient at a layer independent of the activations of the previous layers?

Is the gradient at a layer (of a feed-forward neural network) independent of the activations of the previous layers? Yes, as per @recessive answer they are indeed independent of the previous layers. ...
user avatar
  • 211
1 vote

Is the gradient at a layer independent of the activations of the previous layers?

Yes, this is the premise of back-propagation, the gradient at layer $j_{n}$ is not impacted by the gradient at layer $j_{n-1}$. This allows you to start with a gradient at the output layer and ...
user avatar
  • 1,324
1 vote
Accepted

How is the gradient of the loss function in DQN derived?

In general, if you have a composite function $h(x) = g(f(x))$, then $\frac{dh}{dx} = \frac{d g}{df} \frac{d f}{dx}$. In your case, the function to differentiate is $$L_{i}(\theta_{i}) = \mathbb{E}_{(s,...
user avatar
  • 33k

Only top scored, non community-wiki answers of a minimum length are eligible