Skip to main content
edited title
Link

CS 285 Prof Sergey Levine Lecture, Bounding Derivation for Reinforcement Learning (TRPO)

Source Link

Bounding Derivation for Reinforcement Learning (TRPO)

How can we derive the final result? I can understand the first line, but don't know how the absolute term in the summation is replaced with $2\epsilon t$.

https://www.youtube.com/watch?v=LtAt5M_a0dI&list=PL_iWQOsE6TfXxKgI1GgyV1B_Xa0DxE5eH&index=45 enter image description here

I agree with the $$\sum_{s_t}p_{\theta'}(s_t)f(s_t) \geq \sum_{s_t}\big[p_{\theta}(s_t)f(s_t) - |p_{\theta}(s_t) - p_{\theta'}(s_t)|max_{s_t}f(s_t)\big]$$ $$\because \sum_{s_t}p_{\theta}(s_t)f(s_t) = \sum_{s_t}\big(p_{\theta}(s_t) + p_{\theta'}(s_t) - p_{\theta'}(s_t) \big)f(s_t)$$ $$= \sum_{s_t}p_{\theta'}(s_t)f(s_t) + \big(p_{\theta}(s_t) - p_{\theta'}(s_t) \big)f(s_t)$$ $$ \leq \sum_{s_t}p_{\theta'}(s_t)f(s_t) + \big|p_{\theta}(s_t) - p_{\theta'}(s_t) \big|f(s_t)$$ $$ \leq \sum_{s_t}p_{\theta'}(s_t)f(s_t) + \big|p_{\theta}(s_t) - p_{\theta'}(s_t) \big|max_{s_t}f(s_t)$$

so, $$\sum_{s_t}p_{\theta'}(s_t)f(s_t) \geq \sum_{s_t}\big[p_{\theta}(s_t)f(s_t) - |p_{\theta}(s_t) - p_{\theta'}(s_t)|max_{s_t}f(s_t)\big] \geq \sum_{s_t}\big[p_{\theta}(s_t)f(s_t) - 2\epsilon t max_{s_t}f(s_t)\big] = E_{p_{\theta}(s_t)}[f(s_t)] - \sum_{s_t}2\epsilon t max_{s_t}f(s_t)$$

Here is the question. How does the lecture note remove the summation of the epsilon related term? The professor sasy, since all of them are constant (I understand $max_{s_t}f(s_t)$ is also a constant), we can write like that, but $\sum_{t=1}^{10}(t - 1) = 45 = \sum_{t=1}^{10}t - \sum_{t=1}^{10}1 = 55 - 10 = 45$. We don't write $\sum_{t=1}^{10}1 = 1$ because 1 is a constant.

The abiguous part is there is no concrete size of $s_t$ in the summation, but at least I think just removing the summation in front of the epsilon term is weird.