2
$\begingroup$

Here is the code written by Maxim Lapan. I am reading his book (Deep Reinforcement Learning Hands-on). I have seen a line in his code which is really weird. In the accumulation of the policy gradient $$\partial \theta_{\pi} \gets \partial \theta_{\pi} + \nabla_{\theta}\log\pi_{\theta} (a_i | s_i) (R - V_{\theta}(s_i))$$ we have to compute the advantage $R - V_{\theta}(s_i)$. In line 138, maxim uses adv_v = vals_ref_v - value_v.detach(). Visually, it looks fine, but look at the shape of each term.

ipdb> adv_v.shape                                                                                                                            
torch.Size([128, 128])

ipdb> vals_ref_v.shape                                                                                                                       
torch.Size([128])

ipdb> values_v.detach().shape                                                                                                                
torch.Size([128, 1]) 

In a much simpler code, it is equivalent to

In [1]: import torch                                                            

In [2]: t1 = torch.tensor([1, 2, 3])                                            

In [3]: t2 = torch.tensor([[4], [5], [6]])                                      

In [4]: t1 - t2                                                                 
Out[4]: 
tensor([[-3, -2, -1],
        [-4, -3, -2],
        [-5, -4, -3]])

In [5]: t1 - t2.detach()                                                        
Out[5]: 
tensor([[-3, -2, -1],
        [-4, -3, -2],
        [-5, -4, -3]])

I have trained the agent with his code and it works perfectly fine. I am very confused why it is good practice and what it is doing. Could someone enlighten me on the line adv_v = vals_ref_v - value_v.detach()? For me, the right thing to do was adv_v = vals_ref_v - value_v.squeeze(-1).

Here is the full algorithm used in his book :

UPDATE

enter image description here

As you can see by the image, it is converging even though adv_v = vals_ref_v - value_v.detach() looks wrongly implemented. It is not done yet, but I will update the question later.

$\endgroup$
  • 1
    $\begingroup$ Could you clarify, you trained the agent with his code or with your modified code? If both, did you notice any difference in behaviour? $\endgroup$ – dan888 May 15 at 7:04
  • $\begingroup$ @dan888 With his code. I just cloned the repo. $\endgroup$ – jgauth May 15 at 11:20
  • $\begingroup$ That's interesting. I would say that what you are doing would make more sense as well. I noticed you created an issue on github, I'm curious to see what comes out of that. $\endgroup$ – dan888 May 15 at 13:04
  • $\begingroup$ He used the same line many times in his book, but I can't say why it is working and what it is doing exactly. I am going to update the question in few hours to add a graph of what the mean rewards look like. In his book, maxim said The call to value_v.detach() is important, as we don't want to propagate the PG into our value approximation head. That didn't help me, but maybe it will help someone. $\endgroup$ – jgauth May 15 at 13:22
2
$\begingroup$

Yeah, it seems like it's a wrong implementation. vals_ref_v is a matrix of 1 row, and 128 columns. value_v.detach() is a matrix of 128 row

| improve this answer | |
$\endgroup$
1
$\begingroup$

I changed the line adv_v = vals_ref_v - value_v.detach() to adv_v = vals_ref_v - value_v.squeeze(-1).detach(). It seems the convergence is much faster. According to the A2C algorithm, it is just logic to apply $Q(a, s) - V(s)$, where $Q(a, s)$ and $V(s)$ with the same shape.

The call to detach() is important here as we don't want to propagate the PG into our value approximation head.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.