# What is the best measurement for how good an action of a reinforcement learning agent really is?

Even when we get a valuable reward signal after every single action, this immediate reward only approximates the short term goodness of the action.

To consider the long term effect of an action, we can use the return of an episode, the action value function $$Q(s,a)$$ or the advantage $$A(s,a) = Q(s,a) - V(s)$$. However, these measures do not rate the action in isolation but take all the following actions until the end of an episode into account.

Are there ways to more precisely approximate how good a single action really is considering its short and long term effects?

• You say "And the following actions are sampled from a policy and depend on the state transitions.". The way you sample the actions does not necessarily depend on the state transitions. For example, if you use $\epsilon$-greedy policy to sample an action from the estimate of the value function. Maybe I am misunderstanding your sentence or maybe you didn't express yourself well.
– nbro
Apr 15 '20 at 17:26
• "To encounter the long term effect of an action" does not make sense. I don't think you meant to use the word "encounter"? But I am not quite sure what you are trying to say from the context - it looks like you are saying that the Q value is dependent on the current policy, and you would like some measure of how good an action is independently of the policy? Apr 15 '20 at 21:43
• I've provisionally closed this post until you clarify it by addressing the issues in the comments above. Please, edit your post to clarify it and then vote to re-open it, and I will re-open it.
– nbro
Apr 17 '20 at 13:11
• @nbro, I've removed this part from the question to reduce the chance of confusion. I meant that the actions taken in an episode depend on the policy and the state transitions probabilities. The policy chooses an action based on visited states while the latter determines which states are visited. Apr 18 '20 at 13:17
• @RayWalker What do you mean by "better ways"?
– nbro
Apr 18 '20 at 13:22

If you want to know the effects of an action in e.g. $$n$$ steps ahead, then you can probably formulate this problem as a truncated version of the typical reinforcement learning problem. In practice, you probably can achieve this by changing the discount factor so that the next $$n$$ rewards are more valuable (or are the only ones considered) than the rewards after $$n$$ steps. If you aren't familiar with discount factors, I encourage you to have a look at this concept from a reference book.