TL;DR: read the bold. The rest are details
I am trying to implement Reinforcement Learning:An Introduction, section 13.5 myself:
The algorithm seems to be learning something useful (and not random), as shown in these graphs (different zoom on the same run):
Which show the reward per episode (y axis is the "time alive", x axis is episode number).
However, as can be seen,
The learning does not seem to stabilize.
It looks like every time the reward maxes out (200), it immediately drops.
My relevant code for reference (inspired by pytorch's actor critic)
note: in this question,
xp_batch is ONLY THE VERY LAST (s, a, r, s'), meaning experience replay is not in use in this code!
The actor and critic are both distinct neural networks which
def learn(self, xp_batch):#in this question, xp_batch is ONLY THE VERY LAST (s, a, r, s') for s_t, a_t, r_t, s_t1 in xp_batch: expected_reward_from_t = self.critic_nn(s_t) probs_t = self.actor_nn(s_t) expected_reward_from_t1 = torch.tensor([], dtype=torch.float) if s_t1 is not None: # s_t is not a terminal state, s_t1 exists. expected_reward_from_t1 = self.critic_nn(s_t1) m = Categorical(probs_t) log_prob_t = m.log_prob(a_t) delta = r_t + self.args.gamma * expected_reward_from_t1 - expected_reward_from_t loss_critic = delta * expected_reward_from_t self.critic_optimizer.zero_grad() loss_critic.backward(retain_graph=True) self.critic_optimizer.step() delta.detach() loss_actor = delta * log_prob_t self.actor_optimizer.zero_grad() loss_actor.backward() self.actor_optimizer.step() def select_action(self, state): probs = self.actor_nn(state) m = Categorical(probs) action = m.sample() return action
My questions are:
Am I doing something wrong, or is this to be expected?
I know this can be improved with eligibility traces/experience replay+off policy learning. Before making those upgrades, I want to make sure the current results make sense.