How do we actually sample an action from a policy in policy gradient methods?

Question

Recently I started to look at policy gradient methods and policies are represented as functions with features for larger problems with many states. Many articles and pseudocodes of algorithms mention sampling an action from the policy, but it is unclear to me how.

Actions are something we do in the environment, like going left, right, etc... And functions take some feature values and parameters, make calculations, and 'spit out' some number. So how do we actually map that number to a certain action, and how do we know what action to take?

Answer 1

In those policy gradient algorithms, we typically have:

• Input features $x(s)$ for any state $s$
• A fixed action set $\mathcal{A}$ of a fixed size (in reinforcement learning (RL) it is often assumed that all actions are legal in all states. In some situations, like board games, we'll filter out illegal actions manually in a postprocessing step).
• A function approximator $\pi(x)$ (often a neural network), which takes as input the features $x(s)$ of some state $s$, and produces as output a vector of probabilities, where the vector has a length equal to the size $\vert \mathcal{A} \vert$ of the action space.

Those bolded parts in the last point there are critical for your question. We do not normally have a function that "spits out some number" as you described, we typically have one that "spits out" multiple numbers; one for every action. Often, these numbers can also be directly interpreted as probabilities, because they are forced to all lie in $[0, 1]$ and sum up to $1$ (this is often done using a softmax layer at the end of a neural network for example).

Once you have such a vector of "probabilities", one for every action, it is relatively straightforward to sample from that. Just give every action a probability of being selected equal to its corresponding output.

Note that in some cases, people describe their function approximator as producing "logits" rather than "probabilities" as outputs. These are real-valued outputs which do not necessarily sum up to one, typically produced by a linear layer at the end of a neural network rather than a softmax layer (see e.g. AlphaGo Zero). It is then still implicitly assumed that we convert them into numbers that can be interpreted as probabilities through the application of a softmax, before sampling from it.