What does the notation $\partial \theta_{\pi}$ mean in this actor-critic update rule?

Question

One of the steps in the actor-critic algorithm is $$\partial \theta_{\pi} \gets \partial \theta_{\pi} + \nabla_{\theta}\log\pi_{\theta} (a_i | s_i) (R - V_{\theta}(s_i))$$

For me, $\theta$ are just the weights. Can you explain to me what mean $\partial \theta_{\pi}$?

The whole algorithm comes from Maxim Lepan's book Deep Reinforcement Learning Hands-on page 269.

Here is a picture of the algorithm :

Answer 1

In reinforcement learning, you can distinguish algorithms based on the functions they use to ultimately find the policy (which is the goal in RL anyway!).

• algorithms that attempt to find an optimal value function (an example is Q-learning, which attempts to find a state-action value function), then derive the policy from the value function
• algorithms that directly attempt to find a policy (e.g. REINFORCE and other so-called "policy gradients" algorithms)
• algorithms that use a value function to guide the search for an optimal policy (i.e. actor-critic methods)

More specifically, in actor-critic methods, you have a policy $\pi$ (known as the "actor") and a value function $v$ (known as the "critic"). Hence the name "actor-critic". The idea is that you will use this critic $v$ to "criticize" the policy (or actor) $\pi$, i.e. to guide the search for a good policy. This article 6.6 Actor-Critic Methods (from Sutton and Barto's book) explains the concept quite well.

In your specific example, the policy and value function are assumed to be differentiable (otherwise, you wouldn't be able to compute the derivatives anyway!). They are typically neural networks. $\theta_\pi$ are the parameters of the neural network that represent the policy (i.e. a neural network that receives as input a state and produces as output an action or probability distribution over actions). Similarly, $\theta_v$ are the parameters of the neural network that represent the critic. Then $\partial \theta_\pi$ and $\partial \theta_\pi $ will represent an accumulation of the gradients with respect to the actor and critic, respectively. (The accumulation is probably over the steps from $i=t-1$ to $i = t_{\text{start}}$, but I can't say more because I never implemented actor-critic methods). This should give you an idea, though!