Starting with rewards, states don't have rewards in general. A reward is a number returned at a certain step of the MDP. If you arrange things in sequence over a whole time step $s, a, r, s'$ for state, action, reward, next state, then the reward $r$ is allowed to depend on all three of $s, a, s'$, and it can also be from a random distribution of real numbers, not just a single number.

It is however OK to associate a single number reward with each state, for either leaving that state (when it is $s$ or $s_t$ in the sequence) or arriving in it (when it is $s'$ or $s_{t+1}$). The rewards should be allocated as fits the problem being solved. They are part of the problem definition.

State values are a way to measure longer term benefits of being in a state, and are often something calculated as part of a solution. The formal definition of state value looks like this:

$$v_{\pi}(s) = \mathbb{E}_{\pi}[\sum_{k=0}^{\infty} \gamma^k R_{t+k+1} | S_t=s]$$

In English: "The expected discounted sum of all future rewards when starting from a given state and following a specific policy." The discounted sum is usually called the return or the utility associated with the state.

What is the difference between a reward and a value for a given state?

A state value is composed of many rewards weighted by their probability of occurring in the future. It is a useful summary of possible futures that can be used to help make decisions.

And should the initial values of the states always be set equal to zero?

Not necessarily, but zero is a reasonable default if you have nothing else to go on. Alternatives include:

• Best guesses at true values (perhaps from some previous attempt to solve the problem). This may improvbe speed of convergence depending on how good the guesses are.

• Random values - this may happen if you use a neural network.

• Optimistic values. This is a trick for improving exploration on smaller problems - if you set a value higher than an upper bound on the optimum possible then an agent following an greedy or near-greedy policy will try to reach the associated state at some point, even if other results are already better than a lower default like zero.

Starting with rewards, states don't have rewards in general. A reward is a number returned at a certain step of the MDP. If you arrange things in sequence over a whole time step $s, a, r, s'$ for state, action, reward, next state, then the reward $r$ is allowed to depend on all three of $s, a, s'$, and it can also be from a random distribution of real numbers, not just a single number.

It is however OK to associate a single number reward with each state, for either leaving that state (when it is $s$ or $s_t$ in the sequence) or arriving in it (when it is $s'$ or $s_{t+1}$). The rewards should be allocated as fits the problem being solved. They are part of the problem definition.

State values are a way to measure longer term benefits of being in a state, and are often something calculated as part of a solution. The formal definition of state value looks like this:

$$v_{\pi}(s) = \mathbb{E}_{\pi}[\sum_{k=0}^{\infty} \gamma^k R_{t+k+1} | S_t=s]$$

In English: "The expected discounted sum of all future rewards when starting from a given state and following a specific policy." The discounted sum is usually called the return or the utility associated with the state.

What is the difference between a reward and a value for a given state?

A state value is composed of many rewards weighted by their probability of occurring in the future. It is a useful summary of possible futures that can be used to make decisions.

And should the initial values of the states always be set equal to zero?

Not necessarily, but zero is a reasonable default if you have nothing else to go on. Alternatives include:

• Best guesses at true values (perhaps from some previous attempt to solve the problem). This may improve speed of convergence depending on how good the guesses are.

• Random values - this may happen if you use a neural network.

• Optimistic values. This is a trick for improving exploration on smaller problems - if you set a value higher than an upper bound on the optimum possible then an agent following an greedy or near-greedy policy will try to reach the associated state at some point, even if other results are already better than a lower default like zero.

Starting with rewards, states don't have rewards in general. A reward is a number returned at a certain step of the MDP. If you arrange things in sequence over a whole time step $s, a, r, s'$ for state, action, reward, next state, then the reward $r$ is allowed to depend on all three of $s, a, s'$, and it can also be a distribution of real numbers, not just a single number.

It is however OK to associate a single number reward with each state, for either leaving that state (when it is $s$ or $s_t$ in the sequence) or arriving in it (when it is $s'$ or $s_{t+1}$). The rewards should be allocated as fits the problem being solved. They are part of the problem definition.

State values are a way to measure longer term benefits of being in a state, and are often something calculated as part of a solution. The formal definition of state value looks like this:

$$v_{\pi}(s) = \mathbb{E}_{\pi}[\sum_{k=0}^{\infty} \gamma^k R_{t+k+1} | S_t=s]$$

In English: "The expected discounted sum of all future rewards when starting from a given state and following a specific policy." The discounted sum is usually called the return or the utility associated with the state.

What is the difference between a reward and a value for a given state?

A state value is composed of many rewards weighted by their probability of occurring in the future. It is a useful summary of possible futures that can be used to help make decisions.

Starting with rewards, states don't have rewards in general. A reward is a number returned at a certain step of the MDP. If you arrange things in sequence over a whole time step $s, a, r, s'$ for state, action, reward, next state, then the reward $r$ is allowed to depend on all three of $s, a, s'$, and it can also be from a random distribution of real numbers, not just a single number.

It is however OK to associate a single number reward with each state, for either leaving that state (when it is $s$ or $s_t$ in the sequence) or arriving in it (when it is $s'$ or $s_{t+1}$). The rewards should be allocated as fits the problem being solved. They are part of the problem definition.

State values are a way to measure longer term benefits of being in a state, and are often something calculated as part of a solution. The formal definition of state value looks like this:

$$v_{\pi}(s) = \mathbb{E}_{\pi}[\sum_{k=0}^{\infty} \gamma^k R_{t+k+1} | S_t=s]$$

In English: "The expected discounted sum of all future rewards when starting from a given state and following a specific policy." The discounted sum is usually called the return or the utility associated with the state.

What is the difference between a reward and a value for a given state?

A state value is composed of many rewards weighted by their probability of occurring in the future. It is a useful summary of possible futures that can be used to help make decisions.

And should the initial values of the states always be set equal to zero?

Not necessarily, but zero is a reasonable default if you have nothing else to go on. Alternatives include:

• Best guesses at true values (perhaps from some previous attempt to solve the problem). This may improvbe speed of convergence depending on how good the guesses are.

• Random values - this may happen if you use a neural network.

• Optimistic values. This is a trick for improving exploration on smaller problems - if you set a value higher than an upper bound on the optimum possible then an agent following an greedy or near-greedy policy will try to reach the associated state at some point, even if other results are already better than a lower default like zero.