What does it mean to parameterise a policy in policy gradient methods?

Question

Can you explain policy gradient methods and what it means for the policy to be parameterised? I am reading Sutton and Barto book on reinforcement learning and didn't understand well what it is, can you give some examples?

Answer 1

In the context of RL, for a policy to be parameterised it typically means we explicitly model the policy and is common in policy gradient methods.

Consider value based methods such as Q-learning where our policy is usually something like $\epsilon$-greedy where we choose our action using the following policy

\begin{align} \pi(a|s) = \left\{ \begin{array}{ll} \arg \max_a Q(s,a) & \text{with probability } 1-\epsilon\;; \\ \text{random action} & \text{with probability } \epsilon\;. \end{array}\right. \end{align} Here we have parameterised the policy with $\epsilon$ but the learning is done by learning the Q-functions. When we parameterise a policy we will explicitly model $\pi$ by the following: $$\pi(s|a,\boldsymbol{\theta}) = \mathbb{P}(A_t = a | S_t=s, \boldsymbol{\theta}_t = \boldsymbol{\theta})\;.$$ Learning is now done by learning the parameter $\boldsymbol{\theta}$ that maximise some performance measure $J(\boldsymbol{\theta})$ by doing approximate gradient ascent updates of the form $$\boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_t + \alpha \hat{\Delta J(\boldsymbol{\theta}_t)}. $$

Note that, as per the Sutton and Barto textbook, $\hat{\Delta J(\boldsymbol{\theta}_t)}$ is a noisy, stochastic estimate of $\Delta J(\boldsymbol{\theta}_t)$ where the former approximates the latter in expectation.

The policy can be parameterised in any way as long as it is differentiable with respect to the parameters. Commonly in Deep RL the policy is parameterised as a neural network so $\boldsymbol{\theta}$ would be the weights of the network.