Meaning of Actor Output in Actor Critic Reinforcement Learning

Question

In actor critic, The equations for calculating the loss in actor critic are an

actor loss (parameterized by $\theta$)

$$log[\pi_\theta(s_t,a_t)]Q_w(s_t,a_t)$$

and a critic loss (parameterized by $w$)

$$r(s_t,a_t) + \gamma Q_w(s_{t+1}, a_{t+1}) - Q_w(s_{t}, a_t).$$

This is bootstrapping in experience replay:

$$ L_i(\theta_i) = \mathbb{E}_{(s, a, r, s') \sim U(D)} \left[ \left(r + \gamma \max_{a'} Q(s', a'; \theta_i^-) - Q(s, a; \theta_i)\right)^2 \right] $$

It is clear that bootstrapping is comparable to the critic loss, except that the $max$ operation is lacking from the critic.

As i see it, (correct me if I'm wrong):

$Q(s_t,a_t) = V(s_{t+1}) + r_t$ where $a_t$ is the actual action that had been taken.

The critic, as I understand, estimates $V(s)$

My question:

What exactly is the critic calculating?

What In actor critic outputs $Q(s_{t+1},a_{t+1})$?

It seems to me like the critic calculates the average next state $s_{t+1}$ value, over all possible actions, with their corresponding probabilities, yielding

$Q(s_t, a_t) = r_t + \sum_{a_{t+1} \in A}P(a_{t+1}|s_t)V(s_{t+1})$

Which would mean that in order to get $Q(s_{t+1}, a_{t+1})$ for the above formula, I would need to calculate

$Q(s_{t+1}, a_{t+1}) = r_{t+1} + \sum_{a_{t+2} \in A}P(a_{t+2}|s_{t+1})V(s_{t+2})$

Where $V(s_{t+2})$ is the critic output on $s_{t+2}$, a state we get to by taking action $a_{t+1}$ from state $s_{t+1}$ but I am not sure that is indeed the meaning of the critic output and still it is unclear to me how to get $Q(s_{t+1}, a_{t+1})$ from actor critic.

If indeed that is what's being calculated, then why is it mathematically true that an improvement is being made? Or why does it make sense (even if not mathematically always true)?

---

Practical use:

I want to use actor critic with experience replay in an environment with a large action space (could be continuous). Therefore, I cannot use the $max$ term. I need to understand the correct equation for the critic loss, and why it works.

Answer 1

When using the loss function for the critic described in your question, the Actor-Critic is an on-policy approach (as are most Actor-Critic methods). Your intuition as to what it is learning seems to be quite close, but the notation/terminology is not quite on point.

First it's important to realize that the $Q(s, a)$ critic is an estimator, we're training it to estimate state-action values. You could say that we are training it such that it can hopefully provide accurate estimates of:

$$Q_w^{\pi} (s_t, a_t) \approx \mathbb{E}_{\pi} \left[ r_t + \gamma V^{\pi}(s_{t+1}) \right].$$

You'll notice I've added quite a number of symbols there in comparison to your $Q(s_t, a_t) = r_t + V(s_{t+1})$:

• I have added the $\pi$ superscript to $Q$ and $V$; this denotes the behaviour policy, which is the policy that we're using to generate experience. In on-policy methods, this is equal to the target policy (the policy for which we're learning to predict values). Adding this superscript makes explicit the fact that we're learning expected returns for states and state-action pairs that are only accurate under the assumption that we continue following the $\pi$ policy from state $s_{t+1}$ onwards.
• I added the discount factor $\gamma$, which is probably just a tiny detail you forgot.
• I added $\mathbb{E}_{\pi}$ to indicate that we're trying to estimate an expectation under $\pi$ (and the environment's dynamics).

So, the critic is trained to estimate $Q^{\pi}(s, a)$, which can intuitively be interpreted as the long-term discounted rewards that we expect to collect when executing $a$ in $s$, and selecting actions according to the distribution $\pi$ subsequently. It definitely still is trying to estimate $Q(s, a)$ values for state-action pairs, not just $V(s)$ values for states alone.

---

What In actor critic outputs $Q(s_{t+1},a_{t+1})$?

In practice, when using the loss function described in your question, $a_{t+1}$ really simply is a single action selected in an actual trajectory of experience by the policy $\pi$. The trained network simply takes $s_{t+1}$ as input, and the output corresponding to a single action $a_{t+1}$ as selected by the policy is used as the value for $Q(s_{t+1},a_{t+1})$ in the update rule.

The update rule does not involve any sum over all actions, multiplied with their probabilities. The "trick" is that we do not just run the update rule a single time, but we expect to generate lots (sometimes millions) of trajectories as experience, and we repeatedly run the update rule. In different trajectories, we'll experience the different actions $a_t$ as samples with approximately the correct frequencies, and in expectation we'll have proper update targets (except for potential bias resulting from function approximation).

---

I want to use actor critic with experience replay in an environment with a large action space

The Actor-Critic method in your question is, as I mentioned above, on-policy. This means that the experience used in update rules has to be generated according to exactly the same policy for which you are also learning value estimates. This is incompatible with the idea of experience replay, because old trajectories stored in a replay buffer were generated by older versions of your policy.

There are off-policy Actor-Critic methods which can correct for the mismatch in distributions and use experience replay, but these are going to be quite a bit more complicated. Examples are ACER and IMPALA.