1
$\begingroup$

When reading the following paper(page 4): An Approximate Dynamic Programming Approach for Dual Stochastic Model Predictive Control

I could see that they were able to unnest the minimization's in the bellmans equations due to sampling of the expected values. It would be cumbersome to write down their whole setup so i thought i might understand why with another example:

Consider the value function below:

\begin{align} V^{*}(s_0) &= \min_{a_0 \in A_{s_0}}[r(s_0, a_0) + \sum_{s_1}p(s_1|s_0, a_0)V^{*}(s_1)] \\ &= \min_{a_0 \in A_{s_0}}[r(s_0, a_0) + \sum_{s_1}p(s_1|s_0, a_0)]\min_{a_1 \in A_{s_1}}[r(s_1, a_1) + \sum_{s_2}p(s_2|s_1, a_1)V^{*}(s_2)] \\ &(?\neq, = ?) \min_{a_0 \in A_{s_0}, a_1 \in A_{s_1}}[r(s_0, a_0) + \sum_{s_1}p(s_1|s_0, a_0)][r(s_1, a_1) + \sum_{s_2}p(s_2|s_1, a_1)V^{*}(s_2)] \end{align} when should it be an $\neq$ or an $=$ in the third equation and why, i.e how can sampling break the recursion and allow us to unnest the minimizations?

(in the paper, they argue that since they use samples to approximate the expected value of the cost-to-go/next state value function they are allowed to unnest the minimizations)

$\endgroup$
3

0

You must log in to answer this question.