# Meaning of Actor Output in Actor Critic Reinforcement Learning

In actor critic, The equations for calculating the loss in actor critic are an

actor loss (parameterized by $$\theta$$)

$$log[\pi_\theta(s_t,a_t)]Q_w(s_t,a_t)$$

and a critic loss (parameterized by $$w$$)

$$r(s_t,a_t) + \gamma Q_w(s_{t+1}, a_{t+1}) - Q_w(s_{t}, a_t).$$

$$L_i(\theta_i) = \mathbb{E}_{(s, a, r, s') \sim U(D)} \left[ \left(r + \gamma \max_{a'} Q(s', a'; \theta_i^-) - Q(s, a; \theta_i)\right)^2 \right]$$

It is clear that bootstrapping is comparable to the critic loss, except that the $$max$$ operation is lacking from the critic.

As i see it, (correct me if I'm wrong):

$$Q(s_t,a_t) = V(s_{t+1}) + r_t$$ where $$a_t$$ is the actual action that had been taken.

The critic, as I understand, estimates $$V(s)$$

My question:

What exactly is the critic calculating?

What In actor critic outputs $$Q(s_{t+1},a_{t+1})$$?

It seems to me like the critic calculates the average next state $$s_{t+1}$$ value, over all possible actions, with their corresponding probabilities, yielding

$$Q(s_t, a_t) = r_t + \sum_{a_{t+1} \in A}P(a_{t+1}|s_t)V(s_{t+1})$$

Which would mean that in order to get $$Q(s_{t+1}, a_{t+1})$$ for the above formula, I would need to calculate

$$Q(s_{t+1}, a_{t+1}) = r_{t+1} + \sum_{a_{t+2} \in A}P(a_{t+2}|s_{t+1})V(s_{t+2})$$

Where $$V(s_{t+2})$$ is the critic output on $$s_{t+2}$$, a state we get to by taking action $$a_{t+1}$$ from state $$s_{t+1}$$ but I am not sure that is indeed the meaning of the critic output and still it is unclear to me how to get $$Q(s_{t+1}, a_{t+1})$$ from actor critic.

If indeed that is what's being calculated, then why is it mathematically true that an improvement is being made? Or why does it make sense (even if not mathematically always true)?

Practical use:

I want to use actor critic with experience replay in an environment with a large action space (could be continuous). Therefore, I cannot use the $$max$$ term. I need to understand the correct equation for the critic loss, and why it works.

• It is not very clear your "As i see it, (correct me if I'm wrong)" part. Can you be more precise with your notation? For example, you have not $Q(s, a)$ anywhere in the previous equations. Also, why are you trying to look at Q(s, a) as Q(s)? I've never seen Q(s), but only V(s). – nbro Feb 6 at 15:48
• Furthermore, what is the relation between your "As i see it, (correct me if I'm wrong)" part and your actual question: "What exactly is the critic calculating?"? Or, what does experience replay have to do with this? I think you should ask one question per post, otherwise it is even more confusing. Ask just one question. – nbro Feb 6 at 15:51
• @nbro I made a major edit. Please check again to see if it is clearer now. Also, I am not confident enough about any of my understanding, so even asking the correct questions is difficult. – Gulzar Feb 6 at 16:46

When using the loss function for the critic described in your question, the Actor-Critic is an on-policy approach (as are most Actor-Critic methods). Your intuition as to what it is learning seems to be quite close, but the notation/terminology is not quite on point.

First it's important to realize that the $$Q(s, a)$$ critic is an estimator, we're training it to estimate state-action values. You could say that we are training it such that it can hopefully provide accurate estimates of:

$$Q_w^{\pi} (s_t, a_t) \approx \mathbb{E}_{\pi} \left[ r_t + \gamma V^{\pi}(s_{t+1}) \right].$$

You'll notice I've added quite a number of symbols there in comparison to your $$Q(s_t, a_t) = r_t + V(s_{t+1})$$:

• I have added the $$\pi$$ superscript to $$Q$$ and $$V$$; this denotes the behaviour policy, which is the policy that we're using to generate experience. In on-policy methods, this is equal to the target policy (the policy for which we're learning to predict values). Adding this superscript makes explicit the fact that we're learning expected returns for states and state-action pairs that are only accurate under the assumption that we continue following the $$\pi$$ policy from state $$s_{t+1}$$ onwards.
• I added the discount factor $$\gamma$$, which is probably just a tiny detail you forgot.
• I added $$\mathbb{E}_{\pi}$$ to indicate that we're trying to estimate an expectation under $$\pi$$ (and the environment's dynamics).

So, the critic is trained to estimate $$Q^{\pi}(s, a)$$, which can intuitively be interpreted as the long-term discounted rewards that we expect to collect when executing $$a$$ in $$s$$, and selecting actions according to the distribution $$\pi$$ subsequently. It definitely still is trying to estimate $$Q(s, a)$$ values for state-action pairs, not just $$V(s)$$ values for states alone.

What In actor critic outputs $$Q(s_{t+1},a_{t+1})$$?

In practice, when using the loss function described in your question, $$a_{t+1}$$ really simply is a single action selected in an actual trajectory of experience by the policy $$\pi$$. The trained network simply takes $$s_{t+1}$$ as input, and the output corresponding to a single action $$a_{t+1}$$ as selected by the policy is used as the value for $$Q(s_{t+1},a_{t+1})$$ in the update rule.

The update rule does not involve any sum over all actions, multiplied with their probabilities. The "trick" is that we do not just run the update rule a single time, but we expect to generate lots (sometimes millions) of trajectories as experience, and we repeatedly run the update rule. In different trajectories, we'll experience the different actions $$a_t$$ as samples with approximately the correct frequencies, and in expectation we'll have proper update targets (except for potential bias resulting from function approximation).

I want to use actor critic with experience replay in an environment with a large action space

The Actor-Critic method in your question is, as I mentioned above, on-policy. This means that the experience used in update rules has to be generated according to exactly the same policy for which you are also learning value estimates. This is incompatible with the idea of experience replay, because old trajectories stored in a replay buffer were generated by older versions of your policy.

There are off-policy Actor-Critic methods which can correct for the mismatch in distributions and use experience replay, but these are going to be quite a bit more complicated. Examples are ACER and IMPALA.

• Thanks! Just one question: to my best understanding, the critic is a function mapping from states to rewards, meaning it takes no action as input. For example, openAI's github.com/pytorch/examples/blob/master/reinforcement_learning/… and also medium.freecodecamp.org/… That said, I may just be wrong, and the code to which I linked just calls something "critic" that is actually not a "literature critic" So, what IS a critic? a function from state only, or from state,action to reward? – Gulzar Feb 6 at 18:33
• The first link is to code where the critic outputs a value given a state ("value head"). The second actually shows equations for Q(s,a), but it is unclear where 'a' comes from. Also the drawings show that the Qhat doesn't take an action in, ever. Please expand on how to evaluate the Q(s,a) at t and t+1, and how to calculate the loss, and most importantly, what the critic maps from: states, or states,actions Thanks! – Gulzar Feb 6 at 18:37
• And another confution about the same topic: researchgate.net/profile/Dewen_Hu/publication/6170058/figure/… shows again, the critic doesn't take an action as input. I just have o get the critic definition down – Gulzar Feb 6 at 18:39
• @Gulzar If someone writes the critic as providing $Q(s, a)$ estimates, then it should be exactly that; something that maps from state-action pairs to values (in Deep Neural Networks often implemented as a network that takes a state vector as input, and produces multiple outputs; one per action, which can then be interpreted as the $Q(s, a)$ for the corresponding output). Actor-Critic is not one specific algorithm though. The $Q(s, a)$ can in practice be replaced by many many different things by "subtracting a state-dependent baseline". – Dennis Soemers Feb 6 at 18:42
• The blog post you linked to discusses one such replacement when it starts talking about the advantage function $A(s, a)$ (which is again a mapping from state-action pairs to scalars). And then that blog post also shows how the $A(s, a)$ function can again be estimated using only a trained $V(s)$ estimator + a reward observation $r$. Once you've arrived at that point, it's sufficient to train a neural network to provide only the estimates for $V(s)$, so then you only need a mapping from states to scalars. – Dennis Soemers Feb 6 at 18:44