I'm trying to create a simple Dyna-Q agent to solve small mazes, in python. For the Q function, Q(s,a), I'm just using a matrix, where each row is for a state value, and each column is for one of the 4 actions (up, down, left, right).
I've implemented the "real experience" part, which is basically just straightforward SARSA. It solves a moderately hard (i.e., have to go around a few obstacles) maze in 2000-8000 steps (in the first episode, it will no doubt decrease with more). So I know that part is working reliably.
Now, adding the part that simulates experience based on what it knows of the model to update the Q values more, I'm having trouble. The way I'm doing it is to keep an
experiences list (a lot like experience replay), where each time I take a real action, I add its (S,A,R,S') to that list.
Then, when I want to simulate an experience, I take a random (S,A,R,S') tuple from that list (David Silver mentions in his lecture (#8) on this that you can either update your transition probability matrix P and reward matrix R by changing their values, or just sample from the experience list, which should be equivalent). In my case, with a given S and A, since it's deterministic, R and S' are also going to be the same as the ones I sampled from the tuple. Then I calculate Q(S,A) and max_A'(Q(S',A')), to get the TD error (same as above), and do stochastic gradient descent with it to change Q(S,A) in the right direction.
But it's not working. When I add simulated experiences, it never finds the goal. I've tried poking around to figure out why, and all I can see that's weird is that the Q values continually increase as time goes on (while, without experiences, they settle to correct values).
Does anyone have any advice about things I could try? I've looked at the sampled experiences, the Q values in the experience loop, the gradient, etc... and nothing really sticks out, aside from the Q values growing.
edit: here's the code. The first part (one step TD learning) is working great. Adding the planning loop part screws it up.
def dynaQ(self, N_steps=100, N_plan_steps=5): self.initEpisode() for i in range(N_steps): #Get current state, next action, reward, next state s = self.getStateVec() a = self.epsGreedyAction(s) r, s_next = self.iterate(a) #Get Q values, Q_next is detached so it doesn't get changed by the gradient Q_cur = self.Q[s, a] Q_next = torch.max(self.Q[s_next]).detach().item() TD0_error = (r + self.params['gamma']*Q_next - Q_cur).pow(2).sum() #SGD self.optimizer.zero_grad() TD0_error.backward() self.optimizer.step() #Add to experience buffer e = Experience(s, a, r, s_next) self.updateModel(e) for j in range(N_plan_steps): xp = self.experiences[randint(0,len(self.experiences)-1)] Q_cur0 = self.Q[xp.s, xp.a] Q_next0 = torch.max(self.Q[xp.s_next]).detach().item() TD0_error0 = (xp.r + self.params['gamma']*Q_next0 - Q_cur0).pow(2).sum() self.optimizer.zero_grad() TD0_error0.backward() self.optimizer.step()