I was watching a lecture on policy gradients and Bellman equations. And they say that a Bellman equation indirectly creates a policy, while the policy gradient directly learns a policy. Why is this?
$\begingroup$
$\endgroup$
0
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
3
Policy gradient methods directly learn parameters of a policy function, which is a mapping from states to actions. For example, $p(s, a)$ can denote a function which takes a state $s$ and an action $a$ as input, and returns a probability of taking action $a$ in state $s$ as output (equivalently it could just take $s$ as input, and output a vector or a distribution of probabilities for all actions). Such a function will often have some parameters (for example, a vector of weights if it's a linear function of the state, or matrices of weights in the case of a neural network). Policy gradient methods directly modify the function by modifying (learning) the parameters of that function.
The Bellman equation is typically used to learn a function that computes different kinds of values, such as $Q(s, a)$ values, which can intuitively be understood to denote all the future rewards that we expect to get if we take action $a$ in state $s$ (and follow a certain policy, typically "the optimal policy" afterwards until the end of time). Such a $Q(s, a)$ function that we learn is not a directly learned policy, because it is not a direct mapping from current states to actions, it doesn't directly tell us which action to take.
It can still be used indirectly to create a policy. For example, once such a $Q(s, a)$ function is learned, the greedy policy can be easily implemented by looping over all possible actions, and taking the action that gives the greatest $Q(s, a)$ output. This is indirect because the learned function does not directly give you the actions, but only provides information which you can convert into actions with some extra processing.
answered Dec 18, 2017 at 14:20
-
$\begingroup$ This makes sense. So, in theory: Q-learning is more general? I've heard that policy gradient methods are sample efficent. But Q-Learning methods are more expressive. Would you agree? $\endgroup$– echoCommented Dec 19, 2017 at 0:30
-
$\begingroup$ I suppose you could say Q-learning is more general in the sense that it's off-policy yes, which means that you can follow a certain policy (e.g. epsilon-greedy), and learn the Q-values for a different policy (e.g. the greedy policy). This is not possible with policy gradient methods as far as I'm aware, they're always on-policy, meaning they optimize their actions for their current policy. This typically results in slightly ''safer'' behaviour if you're using an epsilon-greedy policy for example, because it takes into account that it will sometimes select completely random actions $\endgroup$– Dennis Soemers ♦Commented Dec 19, 2017 at 8:38
-
$\begingroup$ Q-learning is indeed typically less sample efficient, because it cannot learn from the n-step return after a sequence of n steps if anywhere within that sequence an action was taken that would not have been taken by the policy you're learning values for. That's in the typical case though, there are ways to improve this (e.g. Retrace(lambda)) $\endgroup$– Dennis Soemers ♦Commented Dec 19, 2017 at 8:39