# What is the best measurement for how good an action of a reinforcement learning agent really is?

Even when we get a valuable reward signal after every single action, this immediate reward only approximates the short term goodness of the action.

To consider the long term effect of an action, we can use the return of an episode, the action value function $$Q(s,a)$$ or the advantage $$A(s,a) = Q(s,a) - V(s)$$. However, these measures do not rate the action in isolation but take all the following actions until the end of an episode into account.

Are there ways to more precisely approximate how good a single action really is considering its short and long term effects?

• You say "And the following actions are sampled from a policy and depend on the state transitions.". The way you sample the actions does not necessarily depend on the state transitions. For example, if you use $\epsilon$-greedy policy to sample an action from the estimate of the value function. Maybe I am misunderstanding your sentence or maybe you didn't express yourself well. – nbro Apr 15 at 17:26
• "To encounter the long term effect of an action" does not make sense. I don't think you meant to use the word "encounter"? But I am not quite sure what you are trying to say from the context - it looks like you are saying that the Q value is dependent on the current policy, and you would like some measure of how good an action is independently of the policy? – Neil Slater Apr 15 at 21:43
• I've provisionally closed this post until you clarify it by addressing the issues in the comments above. Please, edit your post to clarify it and then vote to re-open it, and I will re-open it. – nbro Apr 17 at 13:11
• @nbro, I've removed this part from the question to reduce the chance of confusion. I meant that the actions taken in an episode depend on the policy and the state transitions probabilities. The policy chooses an action based on visited states while the latter determines which states are visited. – Ray Walker Apr 18 at 13:17
• @RayWalker What do you mean by "better ways"? – nbro Apr 18 at 13:22

If you want to know the effects of an action in e.g. $$n$$ steps ahead, then you can probably formulate this problem as a truncated version of the typical reinforcement learning problem. In practice, you probably can achieve this by changing the discount factor so that the next $$n$$ rewards are more valuable (or are the only ones considered) than the rewards after $$n$$ steps. If you aren't familiar with discount factors, I encourage you to have a look at this concept from a reference book.