New answers tagged pytorch
2
Here is the commit
I fixed few minor errors, but the major one was when I saw what the line histories = [deque(maxlen=self.reward_steps)] * len(self.env.envs) was doing. It was just repeating the same queue.
In [2]: histories = [deque(maxlen=5)] * 4
In [3]: histories ...
2
Yeah, it seems like it's a wrong implementation.
vals_ref_v is a matrix of 1 row, and 128 columns.
value_v.detach() is a matrix of 128 row
1
I changed the line adv_v = vals_ref_v - value_v.detach() to adv_v = vals_ref_v - value_v.squeeze(-1).detach(). It seems the convergence is much faster. According to the A2C algorithm, it is just logic to apply $Q(a, s) - V(s)$, where $Q(a, s)$ and $V(s)$ with the same shape.
The call to detach() is important here as we don't want to propagate the PG into ...
0
I don't know if this comment will be helpful, but shouldn't the sum of the log determinant of the Jacobian (LDJ) have opposite sign in the forward and inverse pass? I'm not talking about the LDJ being sum of the positive scaling function in the forward and sum of the negative of the scaling function in the inverse, I'm talking about the LDJ itself.
For ...
0
class ExperienceSource():
def __init__(self, env, agent, reward_steps):
self.env = env
self.agent = agent
self.reward_steps = reward_steps
def __iter__(self):
histories = [deque(maxlen=self.reward_steps) for i in range(len(self.env.envs))]
states = self.env.reset()
while True:
for idx, ...
Top 50 recent answers are included
