What is the difference between a reward and a value for a given state?

Question

I am trying to learn reinforcement learning and I am focusing on the value iteration. I am looking at the example of grid world, and I am trying to implement it in python. While doing this, I encountered the situation in which I had to set the rewards for the agent, but looking at the theory, I have found that each state has also a value, which is found using the value iteration.

So, my doubt is: What is the difference between a reward and a value for a given state? And should the initial values of the states always be set equal to zero?

Answer 1

Starting with rewards, states don't have rewards in general. A reward is a number returned at a certain step of the MDP. If you arrange things in sequence over a whole time step $s, a, r, s'$ for state, action, reward, next state, then the reward $r$ is allowed to depend on all three of $s, a, s'$, and it can also be from a random distribution of real numbers, not just a single number.

It is however OK to associate a single number reward with each state, for either leaving that state (when it is $s$ or $s_t$ in the sequence) or arriving in it (when it is $s'$ or $s_{t+1}$). The rewards should be allocated as fits the problem being solved. They are part of the problem definition.

State values are a way to measure longer term benefits of being in a state, and are often something calculated as part of a solution. The formal definition of state value looks like this:

$$v_{\pi}(s) = \mathbb{E}_{\pi}[\sum_{k=0}^{\infty} \gamma^k R_{t+k+1} | S_t=s]$$

In English: "The expected discounted sum of all future rewards when starting from a given state and following a specific policy." The discounted sum is usually called the return or the utility associated with the state.

What is the difference between a reward and a value for a given state?

A state value is composed of many rewards weighted by their probability of occurring in the future. It is a useful summary of possible futures that can be used to make decisions.

And should the initial values of the states always be set equal to zero?

Not necessarily, but zero is a reasonable default if you have nothing else to go on. Alternatives include:

• Best guesses at true values (perhaps from some previous attempt to solve the problem). This may improve speed of convergence depending on how good the guesses are.

• Random values - this may happen if you use a neural network.

• Optimistic values. This is a trick for improving exploration on smaller problems - if you set a value higher than an upper bound on the optimum possible then an agent following an greedy or near-greedy policy will try to reach the associated state at some point, even if other results are already better than a lower default like zero.

Answer 2

What is the difference between a reward and a value for a given state?

Let us say that an agent took an action from state $A$ and reached state $B$ and got a score $R$. This instantaneous score the agent received on reaching state $B$ is called the reward.

Now, let me introduce you to the concept of return. Assume that an agent followed a particular trajectory:

1. State 1 -> Action 1
2. Reward 1, State 2 -> Action 2
3. Reward 2, State 3 -> Action 3
...
n. Reward n-1, State n (Terminated)

Return (often denoted by $G$) is the sum total of all the rewards obtained by starting from a state State 1 and following a policy.

So, the definition of the return is

$$G(s_1) = R_1 + R_2 + R_3 + ... = \sum_{i=1}^{\infty}R_i$$

Sometimes (most often) these sequences never terminate, so we include a discount factor (Greek letter gamma, $\gamma$) to rewards obtained in the future.

The definition of the discounted return $G$ is

$$G(s_1) = R_1 + \gamma R_2 + \gamma^2 R_3 + ... = \sum_{i=1}^{\infty}\gamma^{i-1} R_i $$

$\gamma$ is a number between $0$ and $1$: it defines how much importance the agent gives to long-term rewards. For a smaller value of $\gamma$, more importance is given for short-term rewards.

Now, coming back to your question. A value of the state is the expected return for an agent starting from that state and following a particular policy. In the case of stochastic policies (policies that have inherent randomness) and/or for environments with stochastic transition probabilities and/or stochastic rewards, the value is the sum of (the returns of all trajectories multiplied by the probability of taking that trajectory).

And should the initial values of the states always be set equal to zero?

Not necessary, zero initialization is one of many ways to initialize. Random initialization is another method. It depends on the environment setting.