Here is the commit
I fixed few minor errors, but the major one was when I saw what the line histories = [deque(maxlen=self.reward_steps)] * len(self.env.envs) was doing. It was just repeating the same queue.
In : histories = [deque(maxlen=5)] * 4
In : histories ...
I changed the line adv_v = vals_ref_v - value_v.detach() to adv_v = vals_ref_v - value_v.squeeze(-1).detach(). It seems the convergence is much faster. According to the A2C algorithm, it is just logic to apply $Q(a, s) - V(s)$, where $Q(a, s)$ and $V(s)$ with the same shape.
The call to detach() is important here as we don't want to propagate the PG into ...
I don't know if this comment will be helpful, but shouldn't the sum of the log determinant of the Jacobian (LDJ) have opposite sign in the forward and inverse pass? I'm not talking about the LDJ being sum of the positive scaling function in the forward and sum of the negative of the scaling function in the inverse, I'm talking about the LDJ itself.
def __init__(self, env, agent, reward_steps):
self.env = env
self.agent = agent
self.reward_steps = reward_steps
histories = [deque(maxlen=self.reward_steps) for i in range(len(self.env.envs))]
states = self.env.reset()
for idx, ...