# What does the notation $\partial \theta_{\pi}$ mean in this actor-critic update rule?

One of the steps in the actor-critic algorithm is $$\partial \theta_{\pi} \gets \partial \theta_{\pi} + \nabla_{\theta}\log\pi_{\theta} (a_i | s_i) (R - V_{\theta}(s_i))$$

For me, $$\theta$$ are just the weights. Can you explain to me what mean $$\partial \theta_{\pi}$$?

The whole algorithm comes from Maxim Lepan's book Deep Reinforcement Learning Hands-on page 269.

Here is a picture of the algorithm : More specifically, in actor-critic methods, you have a policy $$\pi$$ (known as the "actor") and a value function $$v$$ (known as the "critic"). Hence the name "actor-critic". The idea is that you will use this critic $$v$$ to "criticize" the policy (or actor) $$\pi$$, i.e. to guide the search for a good policy. This article 6.6 Actor-Critic Methods (from Sutton and Barto's book) explains the concept quite well.
In your specific example, the policy and value function are assumed to be differentiable (otherwise, you wouldn't be able to compute the derivatives anyway!). They are typically neural networks. $$\theta_\pi$$ are the parameters of the neural network that represent the policy (i.e. a neural network that receives as input a state and produces as output an action or probability distribution over actions). Similarly, $$\theta_v$$ are the parameters of the neural network that represent the critic. Then $$\partial \theta_\pi$$ and $$\partial \theta_\pi$$ will represent an accumulation of the gradients with respect to the actor and critic, respectively. (The accumulation is probably over the steps from $$i=t-1$$ to $$i = t_{\text{start}}$$, but I can't say more because I never implemented actor-critic methods). This should give you an idea, though!