Some RL algorithms (especially policy gradients) initialize with random policies, which often manifests as random jitter on spot for a long time?

Question

I am reviewing a statement on the website for ES regarding structured exploration.

https://blog.openai.com/evolution-strategies/

Structured exploration. Some RL algorithms (especially policy gradients) initialize with random policies, which often manifests as random jitter on spot for a long time. This effect is mitigated in Q-Learning due to epsilon-greedy policies, where the max operation can cause the agents to perform some consistent action for a while (e.g. holding down a left arrow). This is more likely to do something in a game than if the agent jitters on spot, as is the case with policy gradients. Similar to Q-learning, ES does not suffer from these problems because we can use deterministic policies and achieve consistent exploration.

Where can I find sources showing that policy gradients initialize with random policies, whereas Q-Learning uses epsilon-greedy policies?

Also, what does "max operation" have to do with epsilon-greedy policies?

Answer 1

Where can I find sources showing that policy gradients initialize with random policies, whereas Q-Learning uses epsilon-greedy policies?

You can find example algorithms for Q learning and policy gradients in Sutton & Barto's Reinforcement Learning: An Introduction - Q learning is in chapter 6, and policy gradients explained in chapter 13.

Neither of these things are strictly true in all cases. However, both are very common situations for the two kinds of learning agent:

• Policy gradient solvers learn a policy function $\pi(a|s)$ that gives the probability of taking action $a$ given observed state $s$. Typically this is implemented as a neural network. Neural networks are initialised randomly. For discrete action selection using softmax output layer, the initial function will roughly be choosing evenly from all possible actions. So if some actions oppose and undo each other, e.g. move left and move right are options - then the situation as described in your quote can easily happen. Nowadays there are many other solutions using policy gradients that don't suffer as much with this effect. For instance, deterministic policy gradient methods such as A2C or DDPG.

• For Q learning, and many variants of it, $\epsilon$-greedy is a very commonly used action selection mechanism. It is convenient and simple, and allows a simple parameter to control balance between exploration and exploitation whilst learning. However, Q learning can work with any action selection mechanism that has some possibility of acting optimally at least occasionally.

The best approach to action selection during the learning process in both policy-based methods and value-based methods is an active area of research. So if you read the RL literature you may find a lot of variation. The blog you quoted from has identified two quite common choices.

Also, what does "max operation" have to do with epsilon-greedy policies?

The "max operation" is finding the maximum value of some function produced when varying a parameter. The related "argmax operation" is finding the value of the parameter that produces the maximum value. Q learning can use both types of operation, but specifically uses $\text{argmax}$ for $\epsilon$-greedy action selection.

An $\epsilon$-greedy policy requires acting optimally (according to current estimates of value function) with probability $p=1-\epsilon$, and randomly with equal probability of each action with probability $p=\epsilon$

In order to do this, the algorithm usually (with probability $p=1-\epsilon$) needs to know what the current best guess at an optimal solution is. That is the greedy action with respect to the current Q values i.e. $\text{argmax}_a Q(s,a)$ -