In actor critic, [The equations for calculating the loss in actor critic][1] are an 

_actor_ loss (parameterized by $\theta$) 

$$log[\pi_\theta(s_t,a_t)]Q_w(s_t,a_t)$$ 

and a _critic_ loss (parameterized by $w$) 

$$r(s_t,a_t) + \gamma Q_w(s_{t+1}, a_{t+1}) - Q_w(s_{t}, a_t).$$


This is [bootstrapping in experience replay][2]:

$$
L_i(\theta_i) = \mathbb{E}_{(s, a, r, s') \sim U(D)} \left[ \left(r + \gamma \max_{a'} Q(s', a'; \theta_i^-) - Q(s, a; \theta_i)\right)^2 \right]
$$

It is clear that bootstrapping is comparable to the critic loss, except that the $max$ operation is lacking from the critic.

**As i see it, (correct me if I'm wrong)**:

$Q(s_t,a_t) = V(s_{t+1}) + r_t$ where $a_t$ is the actual action that had been taken.

The critic, as I understand, estimates $V(s)$


**My question:**

**What exactly is the critic calculating?**

**What In actor critic outputs $Q(s_{t+1},a_{t+1})$?**


It seems to me like the critic calculates the average next state $s_{t+1}$ value, over all possible actions, with their corresponding probabilities, yielding

$Q(s_t, a_t) = r_t + \sum_{a_{t+1} \in A}P(a_{t+1}|s_t)V(s_{t+1})$

Which would mean that in order to get $Q(s_{t+1}, a_{t+1})$ for the above formula, I would need to calculate

$Q(s_{t+1}, a_{t+1}) = r_{t+1} + \sum_{a_{t+2} \in A}P(a_{t+2}|s_{t+1})V(s_{t+2})$

Where $V(s_{t+2})$ is the critic output on $s_{t+2}$, a state we get to by taking action $a_{t+1}$ from state $s_{t+1}$ but I am not sure that is indeed the meaning of the critic output and still it is unclear to me how to get $Q(s_{t+1}, a_{t+1})$ from actor critic.

If indeed that is what's being calculated, then why is it mathematically true that an improvement is being made? Or why does it make sense (even if not mathematically always true)?
_______
Practical use:

I want to use actor critic with experience replay in an environment with a large action space (could be continuous). Therefore, I cannot use the $max$ term. I need to understand the correct equation for the critic loss, and why it works.



  [1]: https://medium.freecodecamp.org/an-intro-to-advantage-actor-critic-methods-lets-play-sonic-the-hedgehog-86d6240171d
  [2]: https://datascience.stackexchange.com/questions/20535/what-is-experience-replay-and-what-are-its-benefits