# Do policy independent state and action values exist in reinforcement learning?

The state value function $$V(s)$$ is defined as the expected return starting in state $$s$$ and acting according to the current policy $$\pi(a|s)$$ till the end of the episode. The state-action values $$Q(s,a)$$ are similarly dependent on the current policy.

Is it also possible to get a policy independent value of a state or an action? Can the immediate reward $$r(s,a,s')$$ be considered a noisy estimate of the action value?

## 1 Answer

Do policy independent state and action values exist in reinforcement learning?

No. They do not exist, because in order to progress in any MDP and receive any reward - i.e. to get any measure of value - you must take an action. Any consistent means of selecting actions is a policy, and the nature of that policy impacts which transitions and rewards you expect to observe, which in turn affects the expected value. Inconsistent means of selecting actions would have no meaning with respect to "expected future return", they would just be measurements that you made in the past.

The closest you can get to a no-policy definition would be values with respect to "special" policies that apply in general to nearly all MDPs:

• Value functions for the uniformly distributed random policy.

• Value functions for any optimal policy (if there is more than one optimal policy, then all the value functions for them will be equal).

• Value functions for any "inverse optimal" policy - i.e. the policy that has lowest possible return. This one is not so useful, although it exists theoretically.

The first two can be useful measurements of an MDP. Although the uniform random policy might not be the best, it encapsulates the situation where the agent has absolutely no knowledge of the MDP, and can be a baseline for comparison. The optimal value functions are often a target for learning algorithms, and sometimes you can calculate bounds or even exact targets for these independently of the learning process, to measure how well an algorithm performs on some test MDP.

Can the immediate reward $$r(s,a,s')$$ be considered a noisy estimate of the action value?

No. Using that notation of the function, it is typically already the expected immediate reward. It is entirely independent of reward seen in any other transitions or time steps, so is systematically incorrect as an estimate for future return in many cases - the only exception being if you know all future rewards will be precisely $$0$$. So it is an unbiased estimate of an action value if $$s'$$ is a terminal state.

Immediate reward is also a good estimate of an action value if the discount factor $$\gamma = 0$$. However, that requires you to define the problem as solving for immediate rewards only, which is not usually a free choice when trying to optimise the behaviour of the agent.

• So a policy independent action-value contradicts the definition of the RL problem, which aims at optimizing the expected return? The agent has to choose actions that are good in the long run and not in the short term. And setting the discount factor $\gamma$ to zero would optimize for immediate reward which probably will result in a policy not performing well on the task. – Ray Walker Apr 21 '20 at 14:08
• @RayWalker: That is a reasonable summary. I would add that you don't have to attempt to optimise an expected return in RL, but it is probably the most common application. Other goals include might include measuring performance of some fixed policy, or discovering aspects of the model used by some other system (e.g. inferring the reward model that an agent is using by observing its behaviour - part of inverse reinforcement learning) – Neil Slater Apr 21 '20 at 14:59
• Great to learn that we might actually choose another goal to optimize for with reinforcement learning! – Ray Walker Apr 21 '20 at 15:22
• You inspired another question you might find interesting. – Ray Walker Apr 21 '20 at 15:23