Even when we get a valuable reward signal after every single action, this immediate reward only approximates the short term goodness of the action.

To consider the long term effect of an action, we can use the return of an episode, the action value function $Q(s,a)$ or the advantage $A(s,a) = Q(s,a) - V(s)$. However, these measures do not rate the action in isolation but take all the following actions until the end of an episode into account.

Are there ways to more precisely approximate how good a single action really is considering its short and long term effects?

  • $\begingroup$ You say "And the following actions are sampled from a policy and depend on the state transitions.". The way you sample the actions does not necessarily depend on the state transitions. For example, if you use $\epsilon$-greedy policy to sample an action from the estimate of the value function. Maybe I am misunderstanding your sentence or maybe you didn't express yourself well. $\endgroup$ – nbro Apr 15 '20 at 17:26
  • $\begingroup$ "To encounter the long term effect of an action" does not make sense. I don't think you meant to use the word "encounter"? But I am not quite sure what you are trying to say from the context - it looks like you are saying that the Q value is dependent on the current policy, and you would like some measure of how good an action is independently of the policy? $\endgroup$ – Neil Slater Apr 15 '20 at 21:43
  • $\begingroup$ I've provisionally closed this post until you clarify it by addressing the issues in the comments above. Please, edit your post to clarify it and then vote to re-open it, and I will re-open it. $\endgroup$ – nbro Apr 17 '20 at 13:11
  • $\begingroup$ @nbro, I've removed this part from the question to reduce the chance of confusion. I meant that the actions taken in an episode depend on the policy and the state transitions probabilities. The policy chooses an action based on visited states while the latter determines which states are visited. $\endgroup$ – Ray Walker Apr 18 '20 at 13:17
  • $\begingroup$ @RayWalker What do you mean by "better ways"? $\endgroup$ – nbro Apr 18 '20 at 13:22

Are there ways to more precisely approximate how good a single action really is considering its short and long term effects?

To understand the short-term effects of an action, just take each of the available actions from the current state and observe the reward for each of them. The action that gives you the highest immediate reward is the best action. However, note that the reward function may change or could be stochastic. In those cases, you may need to estimate the best action e.g. by executing it multiple times.

If you want to know the action that gives you the highest amount of reward in the long run (i.e. that gives you the highest return), then you can use one of the available RL algorithms, which were invented exactly to solve this problem. Basically, you're asking us what is the best RL algorithm. It depends on the problem, as usual.

If you want to know the effects of an action in e.g. $n$ steps ahead, then you can probably formulate this problem as a truncated version of the typical reinforcement learning problem. In practice, you probably can achieve this by changing the discount factor so that the next $n$ rewards are more valuable (or are the only ones considered) than the rewards after $n$ steps. If you aren't familiar with discount factors, I encourage you to have a look at this concept from a reference book.

Note that, in this answer, I am just trying to give you the idea and intuition behind a possible answer to your question (also because your question isn't really suited to provide more detailed or rigorous answers).

  • 1
    $\begingroup$ Thank you very much for the detailed and very insightful answer, @nbro! $\endgroup$ – Ray Walker Apr 18 '20 at 14:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.