Return to Answer

Provided you have a finite number of states and actions, then there will not be an infinite number of terms. Therefore the state and action spaces need to be discrete and finite before the quote from the book applies.

I am having a hard time understanding how one could solve this system of equations.

There are a few techniques for solving simulteneous equations.

However, what I would probably do is number all the state values from $v_1 = v_\pi(s_1)$ to $v_{N = |\mathcal{S}|} = v_\pi(s_N)$, and write out each line in order:

$$v_1 = w_{1,1} v_1 + w_{1,2} v_2 + w_{1,3} v_3 + ... w_{N,3} v_N + r_1$$

Where $r_1$ is a constant - it is the expected immediate reward when starting from state $1$, but that is not important. It is the constant offset value you get from resolving the sum that is not multiplied by any $v_i$ unknown variable.

You can discover the values of $w_{i,j}$ by expanding the sum in the Bellman equation for each state in turn.

At that point you can build a matrix of the weights, and solve the linear equations by taking the inverse of the matrix.

[from comments] But if the game has no end then theoretically the sum of expected future rewards should be infinite.

The time series definition of $v_{\pi}(s)$:

$$v_{\pi}(s) = \mathbb{E}_{\pi}[\sum_{k=0}^{\infty} \gamma^k R_{t+k+1} | S_t = s]$$

does not appear in the Bellman equation used to establish the linear equations. This is the main benefit of the Bellman equation, it changes the infinite series view of returns into a set of relations that must hold between the value functions.

Provided you have a finite number of states and actions, then there will not be an infinite number of terms. Therefore the state and action spaces need to be discrete and finite before the quote from the book applies.

I am having a hard time understanding how one could solve this system of equations.

There are a few techniques for solving simulteneous equations.

However, what I would probably do is number all the state values from $v_1 = v_\pi(s_1)$ to $v_{N = |\mathcal{S}|} = v_\pi(s_N)$, and write out each line in order:

$$v_1 = w_{1,1} v_1 + w_{1,2} v_2 + w_{1,3} v_3 + ... w_{1,N} v_N + r_1$$

Where $r_1$ is a constant - it is the expected immediate reward when starting from state $1$, but that is not important. It is the constant offset value you get from resolving the sum that is not multiplied by any $v_i$ unknown variable.

You can discover the values of $w_{i,j}$ by expanding the sum in the Bellman equation for each state in turn.

At that point you can build a matrix of the weights, and solve the linear equations by taking the inverse of the matrix.

[from comments] But if the game has no end then theoretically the sum of expected future rewards should be infinite.

The time series definition of $v_{\pi}(s)$:

$$v_{\pi}(s) = \mathbb{E}_{\pi}[\sum_{k=0}^{\infty} \gamma^k R_{t+k+1} | S_t = s]$$

does not appear in the Bellman equation used to establish the linear equations. This is the main benefit of the Bellman equation, it changes the infinite series view of returns into a set of relations that must hold between the value functions of each state.

Provided you have a finite number of states and actions, then there will not be an infinite number of terms. Therefore the state and action spaces need to be discrete and finite before the quote from the book applies.

I am having a hard time understanding how one could solve this system of equations.

There are a few techniques for solving simulteneous equations.

However, what I would probably do is number all the state values from $v_1 = v_\pi(s_1)$ to $v_{N = |\mathcal{S}|} = v_\pi(s_N)$, and write out each line in order:

$$v_1 = w_{1,1} v_1 + w_{1,2} v_2 + w_{1,3} v_3 + ... w_{N,3} v_N$$

You can discover the values of $w_{i,j}$ by expanding the sum in the Bellman equation for each state in turn.

At that point you can build a matrix of the weights, and solve the linear equations by taking the inverse of the matrix.

[from comments] But if the game has no end then theoretically the sum of expected future rewards should be infinite.

The time series definition of $v_{\pi}(s)$:

$$v_{\pi}(s) = \mathbb{E}_{\pi}[\sum_{k=0}^{\infty} \gamma^k R_{t+k+1} | S_t = s]$$

does not appear in the Bellman equation used to establish the linear equations. This is the main benefit of the Bellman equation, it changes the infinite series view of returns into a set of relations that must hold between the value functions.

Provided you have a finite number of states and actions, then there will not be an infinite number of terms. Therefore the state and action spaces need to be discrete and finite before the quote from the book applies.

I am having a hard time understanding how one could solve this system of equations.

There are a few techniques for solving simulteneous equations.

However, what I would probably do is number all the state values from $v_1 = v_\pi(s_1)$ to $v_{N = |\mathcal{S}|} = v_\pi(s_N)$, and write out each line in order:

$$v_1 = w_{1,1} v_1 + w_{1,2} v_2 + w_{1,3} v_3 + ... w_{N,3} v_N + r_1$$

Where $r_1$ is a constant - it is the expected immediate reward when starting from state $1$, but that is not important. It is the constant offset value you get from resolving the sum that is not multiplied by any $v_i$ unknown variable.

You can discover the values of $w_{i,j}$ by expanding the sum in the Bellman equation for each state in turn.

At that point you can build a matrix of the weights, and solve the linear equations by taking the inverse of the matrix.

[from comments] But if the game has no end then theoretically the sum of expected future rewards should be infinite.

The time series definition of $v_{\pi}(s)$:

$$v_{\pi}(s) = \mathbb{E}_{\pi}[\sum_{k=0}^{\infty} \gamma^k R_{t+k+1} | S_t = s]$$

does not appear in the Bellman equation used to establish the linear equations. This is the main benefit of the Bellman equation, it changes the infinite series view of returns into a set of relations that must hold between the value functions.

Provided you have a finite number of states and actions, then there will not be an infinite number of terms. Therefore the state and action spaces need to be discrete and finite before the quote from the book applies.

I am having a hard time understanding how one could solve this system of equations.

There are a few techniques for solving simulteneous equations.

However, what I would probably do is number all the state values from $v_1 = v_\pi(s_1)$ to $v_{N = |\mathcal{S}|} = v_\pi(s_N)$, and write out each line in order:

$$v_1 = w_{1,1} v_1 + w_{1,2} v_2 + w_{1,3} v_3 + ... w_{N,3} v_N$$

You can discover the values of $w_{i,j}$ by expanding the sum in the Bellman equation for each state in turn.

At that point you can build a matrix of the weights, and solve the linear equations by taking the inverse of the matrix.

Provided you have a finite number of states and actions, then there will not be an infinite number of terms. Therefore the state and action spaces need to be discrete and finite before the quote from the book applies.

I am having a hard time understanding how one could solve this system of equations.

There are a few techniques for solving simulteneous equations.

However, what I would probably do is number all the state values from $v_1 = v_\pi(s_1)$ to $v_{N = |\mathcal{S}|} = v_\pi(s_N)$, and write out each line in order:

$$v_1 = w_{1,1} v_1 + w_{1,2} v_2 + w_{1,3} v_3 + ... w_{N,3} v_N$$

You can discover the values of $w_{i,j}$ by expanding the sum in the Bellman equation for each state in turn.

At that point you can build a matrix of the weights, and solve the linear equations by taking the inverse of the matrix.

[from comments] But if the game has no end then theoretically the sum of expected future rewards should be infinite.

The time series definition of $v_{\pi}(s)$:

$$v_{\pi}(s) = \mathbb{E}_{\pi}[\sum_{k=0}^{\infty} \gamma^k R_{t+k+1} | S_t = s]$$

does not appear in the Bellman equation used to establish the linear equations. This is the main benefit of the Bellman equation, it changes the infinite series view of returns into a set of relations that must hold between the value functions.