# Understand the DDPG algorithm in Keras

I'm trying to understand the DDPG algorithm using Keras

I found the site and started analyzing the code, I can't understand 2 things.

The algorithm used to write the code presented on the page

In the algorithm image, updating the critic's network does not require gradient

But the gradient is implemented in the code, why?

with tf.GradientTape() as tape:
target_actions = target_actor(next_state_batch)
y = reward_batch + gamma * target_critic([next_state_batch, target_actions])
critic_value = critic_model([state_batch, action_batch])
critic_loss = tf.math.reduce_mean(tf.math.square(y - critic_value))



The second question is why in the photo of the algorithm when calculating the actor's policy gradient are 2 gradients multiplied by themselves and in the code only one gradient is calculated for the critic's network and it's not multiplied by the second gradient?

with tf.GradientTape() as tape:
actions = actor_model(state_batch)
critic_value = critic_model([state_batch, actions])
# Used -value as we want to maximize the value given
# by the critic for our actions
actor_loss = -tf.math.reduce_mean(critic_value)


• I'm not 100% sure about this but it looks like to me that for the first question "minimizing the loss" means making a gradient descent step, thus we need to calculate the gradient. As for the second question, the algorithm looks like the expanded chain rule to me for the gradient of critic_value(state_batch, actor_model(state_batch)). – Hai Nguyen Jul 13 at 14:21
The answer to your first question is because the line 'update the critic by minimising the loss $$L = \frac{1}{N} \sum_i \left( y_i - Q(s_i, a_i |\theta^Q)\right)^2$$ is implying that you will do this by using a gradient, i.e. you calculate the gradient of the loss wrt the parameters and perform a gradient descent step.