Questions about notation in RL

Question

I have some questions on notation in reinforcement learning, since it seems that different papers, books have some different notation. I am writing a thesis and wanted to have the notation consistent. I base my notationon this book, but then a paper like TRPO has some different kind of notation. My first question is the following - some people write $\pi$ under the expectation which means that they sample the trajectory from the policy

$$ \nabla J(\theta) =\mathbb{E}_{\pi}[\sum_{a \in A} \pi(a|S_t)q_{\pi}(S_t,a)\ln\nabla\pi_{\theta}(a|S_t)] = \mathbb{E}_{\pi}[q_{\pi}(S_t,A_t)\ln\nabla\pi_{\theta}(A_t|S_t)] $$

is this the same as the following though, first sampling from the stationary distribution we obtain after running the policy for some long time:

$$ \nabla J(\theta) = \mathbb{E}_{S_t\sim d^\pi, A_t\sim\pi(\cdot|S_t)}[q_{\pi}(S_t,A_t)\ln\nabla\pi_{\theta}(A_t|S_t)] $$.

Also can I assume that I can use $A_t$ instead of $A$ in all my theorems/definitions - thus adding the depoendence of $t$? Next question is how terms involving rewards are written down, for example this is from the TRO paper, but for me reward is a random variable actually, while they directly define it as a function:

$$ \eta(\pi) = \mathbb{E}{s_0,a_0,...}\left[\sum_{t=0}^{\infty} \gamma^t r(s_t)\right], \text{ where} s_0 \sim \rho_0(s_0), \quad a_t \sim \pi(a_t|s_t), \quad s_{t+1} \sim P(s_{t+1}|s_t,a_t). $$

Following th conventions of the book I am following this is actually the term

$$ \mathbb{E}_{S_t\sim d^\pi, A_t\sim\pi(\cdot|S_t),R_{t+1}\sim R(A_t, S_t)}[\sum_t \gamma^tR_{t+1}(S_t, A_t)] = \sum_{r_{t+1}\in R} r_{t+1}p(r_{t+1}|s_t,a_t)\sum_{s_t \in S} d^\pi(s_t)\sum_{a_t \in A}\pi(a_t,s_t)\sum_{t}\gamma^tr_{t+1}(s_t,a_t) $$

isn't it?

Also I am often confused about using conditional expectations vs unconditional expectations in RL litereature. Sometimes I feel like it is used interchangably - is there a reason for that? But maybe I am overthinking a lot of this. Can I show mathematically that all of those notations that I presented are really the same?

Answer 1

Yes, $S_t \sim d^\pi$ is a nice way of saying that the states are distributed according to the state distribution induced by following $\pi$. Whilst $\pi$ does not directly choose the next state, choosing the action (which is sampled directly from $\pi$) is clearly influencing what the next state will be, so the distribution of visited states will depend upon $\pi$.

Generally you can just add a $t$ subscript but that would be on you to make sure it makes sense. Generally it is omitted to avoid overloading notation.

For the reward, I would first point out that a random variable is a function. But you can write this however is easier for you -- generally there are different ways of writing this, e.g. $r_t, r(s_t, a_t), r(s_t, a_t, s_{t+1})$. The first is probably the most commonly used as it is the cleanest to read but they are all equivalent (e.g. the second term is typically used in place of the final term when the environment dynamics are deterministic and thus $s_{t+1}$ doesn't affect the reward).