# Gradient of boltzmann policy over reward function

I'm struggling with an inverse reinforcement learning problem which seems to appear quite often around the literature, yet I can't find any resources explaining it.

The problem is that of calculating the gradient of a Boltzmann policy distribution over the reward weights theta:

$$\displaystyle\pi(s,a)=\frac{\exp(\beta\cdot Q(s,a|\theta))}{\sum_{a'}\exp(\beta\cdot Q(s,a'|\theta')}$$

The theta are a linear parametrisation of the reward function, such that

$$\displaystyle R = \theta^T\phi(s,a)$$

where phi(s,a) are features of the state space. In the simplest of case, one could take $$\phi_i(s,a) = \delta(s,i)$$, that is the feature space is just an indicator function of the state space.

A lot of algorithms simply state to calculate the gradient, but that doesn't seem that trivial, and I'm not managing to infer from the bits of code I found online

Some of the papers using these kind of methods are

Apprenticeship Learning About Multiple Intentions, Babes-Vroman et al

MAP Inference for Bayesian Inverse Reinforcement Learning, J.Choi

Any help would be greatly appreciated

The main point here is that you can write $$Q(s, a|\theta) = R = \theta^\top \phi(s, a)$$. For more details on this, you can read up on Policy Gradients (Chapter 13) from the 2nd edition of the Sutton and Barto book (in fact the expression you're looking for is equation 13.9)
For simplicity, I'm setting $$\beta=1$$, but you can always put it in once you get the idea. Therefore, the expression for $$\pi(s,a,\theta)$$ (I'm including $$\theta$$ here to make the dependence of $$\pi$$ explicit) is now: $$\pi(s,a,\theta) = \frac{exp(\theta^\top \phi(s, a))}{\sum_{a'} exp(\theta^\top \phi(s, a'))} = \frac{A}{B}$$
Also, it is standard to compute $$\nabla_\theta\log \pi(s,a,\theta)$$ instead of $$\nabla_\theta \pi(s,a,\theta)$$ (although you totally can), so I'll go ahead and do that. To differentiate $$\log \pi(s,a,\theta)$$ with respect to the parameters, $$\theta$$, let's do some calculus: $$\nabla_\theta \log \pi(s,a,\theta) = \nabla_\theta \log \frac{A}{B} = \nabla_\theta \log A - \nabla_\theta \log B \\ = \nabla_\theta \left[ \theta^\top \phi(s,a) \right] - \nabla_\theta \left[ \log \sum_{a'} exp(\theta^\top \phi(s, a')) \right] \\ = \phi(s,a) - \frac{1}{\sum_{a'} exp(\theta^\top \phi(s, a'))} \sum_{b} exp(\theta^\top \phi(s, b)) \phi(s,b) \\ = \phi(s,a) - \sum_{b} \frac{exp(\theta^\top \phi(s, b))}{\sum_{a'} exp(\theta^\top \phi(s, a'))} \phi(s,b) \\ = \phi(s,a) - \sum_{b} \pi(s,b,\theta) \phi(s,b)$$
Therefore, you can write $$\nabla_\theta\log \pi(s,a,\theta) = \phi(s,a) - \sum_{b} \pi(s,b,\theta) \phi(s,b)$$