Say we want to train an agent $A$ in an environment $E$ which provides a continuous loss $L$. That is, we want $A$ to choose its actions $a$ so that it minimizes the mistake it does, i.e., it minimizes the loss $L$.

In an Actor-Critic scenario, if I understood it correctly, the idea of the Critic is to learn to "evaluate" the current state of the environment by learning from rewards/losses. The Actor will then be adjusted so that its performed actions satisfy the Critic.

Now, if the optimal loss is always provided already by $E$, what would even be an argument to use a Critic? Is the whole idea of an Actor-Critic infrastructure in this case just wrong? Is an Actor-Critic approach only reasonable if the reward/loss sparse so that the Critic can learn a better one?

ADDED on 22.12.2023: As an example for further clarification, $A$'s goal could be to adjust settings of various machines in $E$ so that a measurement, e.g., a pressure inside a tank, stays at a desired value all the time. So the loss would just be the deviation from that value. This is what I meant by "continuous, optimal" loss; a signal that constantly says how well $A$ is doing. This is very much unlike the case of an inverted pendulum that only provides a loss of, say, $-1$, if it fell over, and otherwise constantly $0$.

  • $\begingroup$ I'm not sure of the answer here but I would conjecture that A2C still has something to give because although the reward/loss is fixed (I assume this was what was meant by continuous) this is not the Q value of a state. In a grid environment with a -1 reward for each step, this is not the Q of a state. The Q should approach -[steps to goal] + [reward for reaching goal]. $\endgroup$
    – foreverska
    Commented Dec 20, 2023 at 16:16
  • 2
    $\begingroup$ What do you mean exactly by "continuous/optimal loss"? Could you elaborate a bit? $\endgroup$ Commented Dec 20, 2023 at 18:47
  • $\begingroup$ @foreverska The reward/loss is not "fixed". Please see my added things. $\endgroup$
    – Mathy
    Commented Dec 22, 2023 at 11:06
  • $\begingroup$ @LucaAnzalone Just did that. $\endgroup$
    – Mathy
    Commented Dec 22, 2023 at 11:06


You must log in to answer this question.