How to make episode ending “good” in reinforcement learning?

TL;DR: read the bold. The rest are details

I am trying to implement Reinforcement Learning:An Introduction, section 13.5 myself:

The algorithm seems to be learning something useful (and not random), as shown in these graphs (different zoom on the same run):

Which show the reward per episode (y axis is the "time alive", x axis is episode number).

However, as can be seen,

1. The learning does not seem to stabilize.

2. It looks like every time the reward maxes out (200), it immediately drops.

My relevant code for reference (inspired by pytorch's actor critic)

note: in this question, xp_batch is ONLY THE VERY LAST (s, a, r, s'), meaning experience replay is not in use in this code!

The actor and critic are both distinct neural networks which

def learn(self, xp_batch):#in this question, xp_batch is ONLY THE VERY LAST (s, a, r, s')
for s_t, a_t, r_t, s_t1 in xp_batch:
expected_reward_from_t = self.critic_nn(s_t)
probs_t = self.actor_nn(s_t)
expected_reward_from_t1 = torch.tensor([[0]], dtype=torch.float)
if s_t1 is not None:  # s_t is not a terminal state, s_t1 exists.
expected_reward_from_t1 = self.critic_nn(s_t1)

m = Categorical(probs_t)
log_prob_t = m.log_prob(a_t)

delta = r_t + self.args.gamma * expected_reward_from_t1 - expected_reward_from_t

loss_critic = delta * expected_reward_from_t
loss_critic.backward(retain_graph=True)
self.critic_optimizer.step()

delta.detach()
loss_actor = delta * log_prob_t
loss_actor.backward()
self.actor_optimizer.step()

def select_action(self, state):
probs = self.actor_nn(state)
m = Categorical(probs)
action  = m.sample()
return action


My questions are:

1. Am I doing something wrong, or is this to be expected?

2. I know this can be improved with eligibility traces/experience replay+off policy learning. Before making those upgrades, I want to make sure the current results make sense.

• I think that questions related to programming issues are better suited for datascience.stackexchange.com or Stack Overflow. Anyway, I've you tried to run the PyTorch example (which you based your implementation on) and see if it behaves similarly to yours? – nbro Feb 14 '19 at 22:26
• The example I linked to uses full episodes discounted returns. Mine is (supposed to be) using immediate rewards, online, regardless of episodes. just the very last sample. So, they aren't supposed to give similar results, which is why I come to the experts and ask if this makes any sense – Gulzar Feb 14 '19 at 22:40
• @nbro If this is better in datascience.stackexchange.com, I don't mind it being moved, should I ask for it via flagging? Anyway, this question is more about the results (in the graphs) than the code. The code is there for reference. – Gulzar Feb 14 '19 at 22:44
• You could simply copy the text of this question to another question on Data Science SE and then (if you want) delete the question from here. Hopefully, there you will find more help, but I can't guarantee that. – nbro Feb 14 '19 at 22:46
• Anyway, I feel like you're attempting to implement some non-trivial algorithms and trying to combine different techniques without really knowing the details of these algorithms and techniques or fully understanding them. This will make your life harder. I would encourage you to first implement what is really known to work. Then try to formalise what you think might work. Only then you should attempt to implement it. – nbro Feb 14 '19 at 22:57