I'm struggling with an inverse reinforcement learning problem which seems to appear quite often around the literature, yet I can't find any resources explaining it.

The problem is that of calculating the gradient of a Boltzmann policy distribution over the reward weights $\theta$:

$$\displaystyle\pi(s,a)=\frac{\exp(\beta\cdot Q(s,a|\theta))}{\sum_{a'}\exp(\beta\cdot Q(s,a'|\theta')}$$

The $\theta$ are a linear parametrization of the reward function, such that

$$\displaystyle R = \theta^T\phi(s,a)$$

where $\phi(s,a)$ are features of the state space. In the simplest of the case, one could take $\phi_i(s,a) = \delta(s,i)$, that is, the feature space is just an indicator function of the state space.

A lot of algorithms simply state to calculate the gradient, but that doesn't seem that trivial, and I'm not managing to infer from the bits of code I found online.

Some of the papers using this kind of methods are Apprenticeship Learning About Multiple Intentions (2011), by Monica Babes-Vroman et al, and MAP Inference for Bayesian Inverse Reinforcement Learning (2011), by Jaedeug Choi et al.


1 Answer 1


The main point here is that you can write $Q(s, a|\theta) = R = \theta^\top \phi(s, a)$. For more details on this, you can read up on Policy Gradients (Chapter 13) from the 2nd edition of the Sutton and Barto book (in fact the expression you're looking for is equation 13.9)

For simplicity, I'm setting $\beta=1$, but you can always put it in once you get the idea. Therefore, the expression for $\pi(s,a,\theta)$ (I'm including $\theta$ here to make the dependence of $\pi$ explicit) is now: $$ \pi(s,a,\theta) = \frac{exp(\theta^\top \phi(s, a))}{\sum_{a'} exp(\theta^\top \phi(s, a'))} = \frac{A}{B} $$

Also, it is standard to compute $\nabla_\theta\log \pi(s,a,\theta)$ instead of $\nabla_\theta \pi(s,a,\theta)$ (although you totally can), so I'll go ahead and do that. To differentiate $\log \pi(s,a,\theta)$ with respect to the parameters, $\theta$, let's do some calculus: $$ \nabla_\theta \log \pi(s,a,\theta) = \nabla_\theta \log \frac{A}{B} = \nabla_\theta \log A - \nabla_\theta \log B \\ = \nabla_\theta \left[ \theta^\top \phi(s,a) \right] - \nabla_\theta \left[ \log \sum_{a'} exp(\theta^\top \phi(s, a')) \right] \\ = \phi(s,a) - \frac{1}{\sum_{a'} exp(\theta^\top \phi(s, a'))} \sum_{b} exp(\theta^\top \phi(s, b)) \phi(s,b) \\ = \phi(s,a) - \sum_{b} \frac{exp(\theta^\top \phi(s, b))}{\sum_{a'} exp(\theta^\top \phi(s, a'))} \phi(s,b) \\ = \phi(s,a) - \sum_{b} \pi(s,b,\theta) \phi(s,b) $$

Therefore, you can write $\nabla_\theta\log \pi(s,a,\theta) = \phi(s,a) - \sum_{b} \pi(s,b,\theta) \phi(s,b)$

  • $\begingroup$ I completely forgot about this question, managed to understand it a few days later exactly the way you explained, so it's great for future people to have this answer public and accepted! Thanks for the contribution $\endgroup$
    – Dominus
    Dec 15, 2018 at 20:49

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .