# Dyna-Q algorithm, having trouble when adding the simulated experiences

I'm trying to create a simple Dyna-Q agent to solve small mazes, in python. For the Q function, Q(s,a), I'm just using a matrix, where each row is for a state value, and each column is for one of the 4 actions (up, down, left, right).

I've implemented the "real experience" part, which is basically just straightforward SARSA. It solves a moderately hard (i.e., have to go around a few obstacles) maze in 2000-8000 steps (in the first episode, it will no doubt decrease with more). So I know that part is working reliably.

Now, adding the part that simulates experience based on what it knows of the model to update the Q values more, I'm having trouble. The way I'm doing it is to keep an experiences list (a lot like experience replay), where each time I take a real action, I add its (S,A,R,S') to that list.

Then, when I want to simulate an experience, I take a random (S,A,R,S') tuple from that list (David Silver mentions in his lecture (#8) on this that you can either update your transition probability matrix P and reward matrix R by changing their values, or just sample from the experience list, which should be equivalent). In my case, with a given S and A, since it's deterministic, R and S' are also going to be the same as the ones I sampled from the tuple. Then I calculate Q(S,A) and max_A'(Q(S',A')), to get the TD error (same as above), and do stochastic gradient descent with it to change Q(S,A) in the right direction.

But it's not working. When I add simulated experiences, it never finds the goal. I've tried poking around to figure out why, and all I can see that's weird is that the Q values continually increase as time goes on (while, without experiences, they settle to correct values).

Does anyone have any advice about things I could try? I've looked at the sampled experiences, the Q values in the experience loop, the gradient, etc... and nothing really sticks out, aside from the Q values growing.

edit: here's the code. The first part (one step TD learning) is working great. Adding the planning loop part screws it up.

def dynaQ(self, N_steps=100, N_plan_steps=5):

self.initEpisode()
for i in range(N_steps):
#Get current state, next action, reward, next state
s = self.getStateVec()
a = self.epsGreedyAction(s)
r, s_next = self.iterate(a)
#Get Q values, Q_next is detached so it doesn't get changed by the gradient
Q_cur = self.Q[s, a]
Q_next = torch.max(self.Q[s_next]).detach().item()
TD0_error = (r + self.params['gamma']*Q_next - Q_cur).pow(2).sum()
#SGD
TD0_error.backward()
self.optimizer.step()
e = Experience(s, a, r, s_next)
self.updateModel(e)

for j in range(N_plan_steps):

xp = self.experiences[randint(0,len(self.experiences)-1)]
Q_cur0 = self.Q[xp.s, xp.a]
Q_next0 = torch.max(self.Q[xp.s_next]).detach().item()
TD0_error0 = (xp.r + self.params['gamma']*Q_next0 - Q_cur0).pow(2).sum()


• I have looked through the code, and cannot see any obvious mistake, although I am only just a beginner with PyTorch. It's a bit unusual to be using full gradient descent with loss functions etc on the tabular version of Q learning. Have you tried just doing self.Q[x.s, x.a] += self.params['alpha'] * (xp.r + self.params['gamma']*Q_next0 - Q_cur0) ? That's usually how tabular DynaQ would be formulated . . . – Neil Slater Oct 11 '18 at 11:39
• @NeilSlater I'm using RMSProp. I don't pass it any parameters besides the Q matrix to optimize. I just do self.optimizer = optim.RMSProp([self.Q]). – GrundleMoof Oct 11 '18 at 17:24