# Should the policy parameters be updated at each time step or at the end of the episode in REINFORCE?

REINFORCE is a Monte Carlo policy gradient algorithm, which updates weights (parameters) of policy network by generating episodes. Here's a pseudo-code from Sutton's book (which is same as the equation in Silver's RL note):

When I try to implement this with my own problem, I found something strange. Here's implementation from Pytorch's official GitHub:

def finish_episode():
R = 0
policy_loss = []
returns = []
for r in policy.rewards[::-1]:
R = r + args.gamma * R
returns.insert(0, R)
returns = torch.tensor(returns)
returns = (returns - returns.mean()) / (returns.std() + eps)
for log_prob, R in zip(policy.saved_log_probs, returns):
policy_loss.append(-log_prob * R)
policy_loss = torch.cat(policy_loss).sum()
policy_loss.backward()
optimizer.step()
del policy.rewards[:]
del policy.saved_log_probs[:]


I feel like there's a difference between the above two. In Sutton's pseudo-code, the algorithm updates $$\theta$$ for each step $$t$$, while the second code (PyTorch's one) accumulate loss and update $$\theta$$ with the summation, i.e. after each episode. I tried to search other implementation of REINFORCE, and I found that most of the implementations follow the second form, update after each generated episodes.

To check whether both give the same result, I changed the second code as

def finish_episode():
R = 0
policy_loss = []
returns = []
for r in policy.rewards[::-1]:
R = r + args.gamma * R
returns.insert(0, R)
returns = torch.tensor(returns)
returns = (returns - returns.mean()) / (returns.std() + eps)
for log_prob, R in zip(policy.saved_log_probs, returns):
loss = -log_prob * R
loss.backward()
optimizer.step()

...


and run it, which gives different result (if my code has no problem). So they are not the same, and I think the last one is more close to the original pseudo-code of REINFORCE. What am I missing now? Is it okay because the results are approximately same? (I'm not sure about this claim)

However, in some sense, I think Pytorch's implementation is the right version of REINFORCE. In Sutton's pseudo-code, episode is generated first, so I think $$\theta$$ shouldn't be updated at each step and should be updated after the total loss is computed. If $$\theta$$ is updated at each step, then such $$\theta$$ might be different with the original $$\theta$$ that used to generate the episode.