Artificial Intelligence Stack Exchange is a question and answer site for people interested in conceptual questions about life and challenges in a world where "cognitive" functions can be mimicked in purely digital environment. It only takes a minute to sign up.
Sign up to join this community
The state value function $V(s)$ is defined as the expected return starting in state $s$ and acting according to the current policy $\pi(a|s)$ till the end of the episode. The state-action values $Q(s,a)$ are similarly dependent on the current policy.
Is it also possible to get a policy independent value of a state or an action? Can the immediate reward $r(s,a,s')$ be considered a noisy estimate of the action value?
Do policy independent state and action values exist in reinforcement learning?
No. They do not exist, because in order to progress in any MDP and receive any reward - i.e. to get any measure of value - you must take an action. Any consistent means of selecting actions is a policy, and the nature of that policy impacts which transitions and rewards you expect to observe, which in turn affects the expected value. Inconsistent means of selecting actions would have no meaning with respect to "expected future return", they would just be measurements that you made in the past.
The closest you can get to a no-policy definition would be values with respect to "special" policies that apply in general to nearly all MDPs:
Value functions for the uniformly distributed random policy.
Value functions for any optimal policy (if there is more than one optimal policy, then all the value functions for them will be equal).
Value functions for any "inverse optimal" policy - i.e. the policy that has lowest possible return. This one is not so useful, although it exists theoretically.
The first two can be useful measurements of an MDP. Although the uniform random policy might not be the best, it encapsulates the situation where the agent has absolutely no knowledge of the MDP, and can be a baseline for comparison. The optimal value functions are often a target for learning algorithms, and sometimes you can calculate bounds or even exact targets for these independently of the learning process, to measure how well an algorithm performs on some test MDP.
Can the immediate reward $r(s,a,s')$ be considered a noisy estimate of the action value?
No. Using that notation of the function, it is typically already the expected immediate reward. It is entirely independent of reward seen in any other transitions or time steps, so is systematically incorrect as an estimate for future return in many cases - the only exception being if you know all future rewards will be precisely $0$. So it is an unbiased estimate of an action value if $s'$ is a terminal state.
Immediate reward is also a good estimate of an action value if the discount factor $\gamma = 0$. However, that requires you to define the problem as solving for immediate rewards only, which is not usually a free choice when trying to optimise the behaviour of the agent.
