2
$\begingroup$

I'm struggling with an inverse reinforcement learning problem which seems to appear quite often around the literature, yet I can't find any resources explaining it.

The problem is that of calculating the gradient of a Boltzmann policy distribution over the reward weights $\theta$:

$$\displaystyle\pi(s,a)=\frac{\exp(\beta\cdot Q(s,a|\theta))}{\sum_{a'}\exp(\beta\cdot Q(s,a'|\theta')}$$

The $\theta$ are a linear parametrization of the reward function, such that

$$\displaystyle R = \theta^T\phi(s,a)$$

where $\phi(s,a)$ are features of the state space. In the simplest of the case, one could take $\phi_i(s,a) = \delta(s,i)$, that is, the feature space is just an indicator function of the state space.

A lot of algorithms simply state to calculate the gradient, but that doesn't seem that trivial, and I'm not managing to infer from the bits of code I found online.

Some of the papers using this kind of methods are Apprenticeship Learning About Multiple Intentions (2011), by Monica Babes-Vroman et al, and MAP Inference for Bayesian Inverse Reinforcement Learning (2011), by Jaedeug Choi et al.

$\endgroup$
2
$\begingroup$

The main point here is that you can write $Q(s, a|\theta) = R = \theta^\top \phi(s, a)$. For more details on this, you can read up on Policy Gradients (Chapter 13) from the 2nd edition of the Sutton and Barto book (in fact the expression you're looking for is equation 13.9)

For simplicity, I'm setting $\beta=1$, but you can always put it in once you get the idea. Therefore, the expression for $\pi(s,a,\theta)$ (I'm including $\theta$ here to make the dependence of $\pi$ explicit) is now: $$ \pi(s,a,\theta) = \frac{exp(\theta^\top \phi(s, a))}{\sum_{a'} exp(\theta^\top \phi(s, a'))} = \frac{A}{B} $$

Also, it is standard to compute $\nabla_\theta\log \pi(s,a,\theta)$ instead of $\nabla_\theta \pi(s,a,\theta)$ (although you totally can), so I'll go ahead and do that. To differentiate $\log \pi(s,a,\theta)$ with respect to the parameters, $\theta$, let's do some calculus: $$ \nabla_\theta \log \pi(s,a,\theta) = \nabla_\theta \log \frac{A}{B} = \nabla_\theta \log A - \nabla_\theta \log B \\ = \nabla_\theta \left[ \theta^\top \phi(s,a) \right] - \nabla_\theta \left[ \log \sum_{a'} exp(\theta^\top \phi(s, a')) \right] \\ = \phi(s,a) - \frac{1}{\sum_{a'} exp(\theta^\top \phi(s, a'))} \sum_{b} exp(\theta^\top \phi(s, b)) \phi(s,b) \\ = \phi(s,a) - \sum_{b} \frac{exp(\theta^\top \phi(s, b))}{\sum_{a'} exp(\theta^\top \phi(s, a'))} \phi(s,b) \\ = \phi(s,a) - \sum_{b} \pi(s,b,\theta) \phi(s,b) $$

Therefore, you can write $\nabla_\theta\log \pi(s,a,\theta) = \phi(s,a) - \sum_{b} \pi(s,b,\theta) \phi(s,b)$

$\endgroup$
  • $\begingroup$ I completely forgot about this question, managed to understand it a few days later exactly the way you explained, so it's great for future people to have this answer public and accepted! Thanks for the contribution $\endgroup$ – Dominus Dec 15 '18 at 20:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.