From S&L's value iteration section:

For arbitrary $v_0$, the sequence $\{v_k\}$ can be shown to converge to $v_*$ under the same conditions that guarantee the existence of $v_*$... value iteration is by reference to the Bellman optimality equation (4.1). Note that value iteration is obtained simply by turning the Bellman optimality equation into an update rule.

Therefore to derive value iteration from policy iteration as its extremely simplified case in a step-by-step manner as requested in your updated question is to formally express the above quoted "turning the Bellman optimality equation into an update rule". And indeed, as you commented, value iteration first does policy improvement and then policy evaluation (normally we have first policy evaluation and then policy improvement in GPI framework of RL). The key here is value iteration can only be rigorously derived from Bellman optimality equation (BOE), not from policy improvement theorem. And once you do the derivation you'll understand there's no conflict as complained in your comment.

To derive value iteration from policy iteration stopped with one sweep, see the same reference section:

If policy evaluation is done iteratively, then convergence exactly to $v_{\pi}$ occurs only in the limit. Must we wait for exact convergence, or can we stop short of that? The example in Figure 4.1 certainly suggests that it may be possible to truncate policy evaluation.

So first see BOE (4.1) as follows which is already very close to your own quoted first equation: $$v_*(s)=\max_{a \in \mathcal{A}}\mathbb{E} \Big(R_{t+1}+\gamma v_*(S_{t+1}) \big|S_t=s, A_t=a\Big)$$

To derive a practical algo to solve for optimal state value $v_*$ in BOE is simply to turn it into an iterative update rule as hinted above in S&B's book. So formally you just simply need to replace LHS's $v_*$ with $v_{k+1}$ and RHS's $v_*$ with $v_k$ which is exactly your requested equation! Your second equation is just further calculating the expectation statistics with model-based RL.

Finally note the $v_k, v_{k+1}$ in your value iteration are not state values themselves because there's no policy evaluation based on state value here at all, thus you cannot rely on policy improvement theorem to prove convergence. To prove convergence of value iteration as hinted above you need BOE and its contraction mapping fixed point theorem to step-by-step prove the uniqueness and convergence of its iterative solution which is entirely another question. There's a diagram from Zhao's Mathematical Foundation of Reinforcement Learning to illustrate the non-conflict relation between value iteration and policy iteration as follows:

And the same reference specifically warned:

One problem of value iteration that may confuse beginners is whether $v_k$ is a state value. The answer is no although $v_k$ converges to the optimal state value $v_*$ eventually. That is because $v_k$ is not guaranteed to satisfy any Bellman equation... r. This might be one of the most confusing problems for beginners studying this algorithm. It is easier if we simply treat them as intermediate variables that emerge when solving the BOE.

That's why strictly speaking one shouldn't say policy improve (PI) in value iteration since there's no policy improvement theorem without state values evaluation (note all $v_k$'s in value iteration has no policy $\pi$ compared to policy iteration), one should only say policy update (PU) in value iteration though it's also consistent with GPI (generalized policy iteration) as shown in above diagram.

From S&L's value iteration section:

For arbitrary $v_0$, the sequence $\{v_k\}$ can be shown to converge to $v_*$ under the same conditions that guarantee the existence of $v_*$... value iteration is by reference to the Bellman optimality equation (4.1). Note that value iteration is obtained simply by turning the Bellman optimality equation into an update rule.

Therefore to derive value iteration from policy iteration as its extremely simplified case in a step-by-step manner as requested in your updated question is to formally express the above quoted "turning the Bellman optimality equation into an update rule". And indeed, as you commented, value iteration first does policy improvement and then policy evaluation (normally we have first policy evaluation and then policy improvement in GPI framework of RL). The key here is value iteration can only be rigorously derived from Bellman optimality equation (BOE), not from policy improvement theorem. And once you do the derivation you'll understand there's no conflict as complained in your comment.

So first see BOE (4.1) as follows which is already very close to your own quoted first equation: $$v_*(s)=\max_{a \in \mathcal{A}}\mathbb{E} \Big(R_{t+1}+\gamma v_*(S_{t+1}) \big|S_t=s, A_t=a\Big)$$

To derive a practical algo to solve for optimal state value $v_*$ in BOE is simply to turn it into an iterative update rule as hinted above in S&B's book. So formally you just simply need to replace LHS's $v_*$ with $v_{k+1}$ and RHS's $v_*$ with $v_k$ which is exactly your requested equation! Your second equation is just further calculating the expectation statistics with model-based RL.

Finally note the $v_k, v_{k+1}$ in your value iteration are not state values themselves because there's no policy evaluation based on state value here at all, thus you cannot rely on policy improvement theorem to prove convergence. To prove convergence of value iteration as hinted above you need BOE and its contraction mapping fixed point theorem to step-by-step prove the uniqueness and convergence of its iterative solution which is entirely another question. There's a diagram from Zhao's Mathematical Foundation of Reinforcement Learning to illustrate the non-conflict relation between value iteration and policy iteration as follows:

And the same reference specifically warned:

One problem of value iteration that may confuse beginners is whether $v_k$ is a state value. The answer is no although $v_k$ converges to the optimal state value $v_*$ eventually. That is because $v_k$ is not guaranteed to satisfy any Bellman equation... This might be one of the most confusing problems for beginners studying this algorithm. It is easier if we simply treat them as intermediate variables that emerge when solving the BOE.

That's why strictly speaking one shouldn't say policy improve (PI) in value iteration since there's no policy improvement theorem without state values evaluation (note all $v_k$'s in value iteration has no subscript $\pi$ compared to policy iteration), one should only say policy update (PU) in value iteration though it's also consistent with GPI (generalized policy iteration) as shown in above diagram.

In fact, the policy evaluation step of policy iteration can be truncated in several ways without losing the convergence guarantees of policy iteration. One important special case is when policy evaluation is stopped after just one sweep. This algorithm is called value iteration... For arbitrary $v_0$, the sequence $\{v_k\}$ can be shown to converge to $v_*$ under the same conditions that guarantee the existence of $v_*$.

Another way of understanding value iteration is by reference to the Bellman optimality equation (4.1). Note that value iteration is obtained simply by turning the Bellman optimality equation into an update rule.

Therefore to derive value iteration from policy iteration with one sweep as its extremely simplified case and to derive convergence of value iteration are two different questions. Please note the $v_k, v_{k+1}$ in your value iteration are not state values themselves because as mentioned above policy evaluation is stopped after just one sweep here, thus you can no longer rely on policy improvement theorem of its previous section to prove convergence. To prove convergence of value iteration as hinted above you need Bellman optimality equation and its contraction mapping fixed point theorem to rigorously prove the uniqueness and convergence of its iterative solution which is another question.

Back to your question to simply derive value iteration from policy iteration with one sweep, see the same reference section:

Also note how the value iteration update is identical to the policy evaluation update (4.5) except that it requires the maximum to be taken over all actions.

Note based on Bellman equation(4.4), (4.5) is an approximated iterative update definition of the same $v_k, v_{k+1}$ in value iteration which are not state values themselves. So value iteration is primarily just 'derived' from (4.5), and then combined with the policy update step which is expressed with the max operator as shown in your question and confirmed by the same section reference:

It can be written as a particularly simple update operation that combines the policy improvement and truncated policy evaluation steps (4.10)

For arbitrary $v_0$, the sequence $\{v_k\}$ can be shown to converge to $v_*$ under the same conditions that guarantee the existence of $v_*$... value iteration is by reference to the Bellman optimality equation (4.1). Note that value iteration is obtained simply by turning the Bellman optimality equation into an update rule.

Therefore to derive value iteration from policy iteration as its extremely simplified case in a step-by-step manner as requested in your updated question is to formally express the above quoted "turning the Bellman optimality equation into an update rule". And indeed, as you commented, value iteration first does policy improvement and then policy evaluation (normally we have first policy evaluation and then policy improvement in GPI framework of RL). The key here is value iteration can only be rigorously derived from Bellman optimality equation (BOE), not from policy improvement theorem. And once you do the derivation you'll understand there's no conflict as complained in your comment.

To derive value iteration from policy iteration stopped with one sweep, see the same reference section:

If policy evaluation is done iteratively, then convergence exactly to $v_{\pi}$ occurs only in the limit. Must we wait for exact convergence, or can we stop short of that? The example in Figure 4.1 certainly suggests that it may be possible to truncate policy evaluation.

So first see BOE (4.1) as follows which is already very close to your own quoted first equation: $$v_*(s)=\max_{a \in \mathcal{A}}\mathbb{E} \Big(R_{t+1}+\gamma v_*(S_{t+1}) \big|S_t=s, A_t=a\Big)$$

To derive a practical algo to solve for optimal state value $v_*$ in BOE is simply to turn it into an iterative update rule as hinted above in S&B's book. So formally you just simply need to replace LHS's $v_*$ with $v_{k+1}$ and RHS's $v_*$ with $v_k$ which is exactly your requested equation! Your second equation is just further calculating the expectation statistics with model-based RL.

Finally note the $v_k, v_{k+1}$ in your value iteration are not state values themselves because there's no policy evaluation based on state value here at all, thus you cannot rely on policy improvement theorem to prove convergence. To prove convergence of value iteration as hinted above you need BOE and its contraction mapping fixed point theorem to step-by-step prove the uniqueness and convergence of its iterative solution which is entirely another question. There's a diagram from Zhao's Mathematical Foundation of Reinforcement Learning to illustrate the non-conflict relation between value iteration and policy iteration as follows:

And the same reference specifically warned:

One problem of value iteration that may confuse beginners is whether $v_k$ is a state value. The answer is no although $v_k$ converges to the optimal state value $v_*$ eventually. That is because $v_k$ is not guaranteed to satisfy any Bellman equation... r. This might be one of the most confusing problems for beginners studying this algorithm. It is easier if we simply treat them as intermediate variables that emerge when solving the BOE.

That's why strictly speaking one shouldn't say policy improve (PI) in value iteration since there's no policy improvement theorem without state values evaluation (note all $v_k$'s in value iteration has no policy $\pi$ compared to policy iteration), one should only say policy update (PU) in value iteration though it's also consistent with GPI (generalized policy iteration) as shown in above diagram.