# How and whether to apply Reinforcement Learning in an Environment with a precise and always available Evaluation?

Say we want to train an agent $$A$$ in an environment $$E$$ which provides a continuous loss $$L$$. That is, we want $$A$$ to choose its actions $$a$$ so that it minimizes the mistake it does, i.e., it minimizes the loss $$L$$.

In an Actor-Critic scenario, if I understood it correctly, the idea of the Critic is to learn to "evaluate" the current state of the environment by learning from rewards/losses. The Actor will then be adjusted so that its performed actions satisfy the Critic.

Now, if the optimal loss is always provided already by $$E$$, what would even be an argument to use a Critic? Is the whole idea of an Actor-Critic infrastructure in this case just wrong? Is an Actor-Critic approach only reasonable if the reward/loss sparse so that the Critic can learn a better one?

ADDED on 22.12.2023: As an example for further clarification, $$A$$'s goal could be to adjust settings of various machines in $$E$$ so that a measurement, e.g., a pressure inside a tank, stays at a desired value all the time. So the loss would just be the deviation from that value. This is what I meant by "continuous, optimal" loss; a signal that constantly says how well $$A$$ is doing. This is very much unlike the case of an inverted pendulum that only provides a loss of, say, $$-1$$, if it fell over, and otherwise constantly $$0$$.

• I'm not sure of the answer here but I would conjecture that A2C still has something to give because although the reward/loss is fixed (I assume this was what was meant by continuous) this is not the Q value of a state. In a grid environment with a -1 reward for each step, this is not the Q of a state. The Q should approach -[steps to goal] + [reward for reaching goal]. Commented Dec 20, 2023 at 16:16
• What do you mean exactly by "continuous/optimal loss"? Could you elaborate a bit? Commented Dec 20, 2023 at 18:47
• @foreverska The reward/loss is not "fixed". Please see my added things. Commented Dec 22, 2023 at 11:06
• @LucaAnzalone Just did that. Commented Dec 22, 2023 at 11:06