Deep Q Learning for Simple Game Not Effective

Question

This is a follow-up question about one I asked earlier. The first question is here. Basically, I have a game where a paddle moves left and right to catch as much "food" as possible. Some food is good (gain points) and some is bad (lose points). NN Architecture:

    #inputs - paddle.x, food.x, food.y, food.type
    #moves: left, right, stay
    model = Sequential()
    model.add(Dense(10, input_shape=(4,), activation='relu'))
    model.add(Dense(10, activation='relu'))
    model.add(Dense(3, activation='linear'))
    model.compile(loss='mean_squared_error', optimizer='adam')

As suggested in the other question, I scaled my inputs to be between 0 and 1. Also, implemented experience replay (although I am not confident I did it correctly).

Here is my ReplayMemory class:

class ReplayMemory():
    def __init__(self, capacity):
        self.capacity = capacity
        self.memory = []
        self.count = 0

    def push(self, experience):
        if len(self.memory) < self.capacity:
            self.memory.append(experience)
        else:
            self.memory[self.count % self.capacity] = experience
        self.count += 1

    def sample(self, batch_size):
        return random.sample(self.memory, batch_size)

    def can_provide_sample(self, batch_size):
        return len(self.memory) >= batch_size

This basically stores states/rewards/actions and returns a random group when asked.

Lastly, here is my learning code:

def learning(num_episodes=20):
        global scores, experiences, target_vecs
        y = 0.8
        eps = 0
        decay_factor = 0.9999

for i in range(num_episodes):
    state = GAME.reset()
    GAME.done = False
    done = False
    counter = 0
    while not done:
        eps *= decay_factor
        counter+=1

        if np.random.random() < eps:
            a = np.random.randint(0, 2)
        else:
            a = np.argmax(model.predict(np.array([scale(state)])))

        new_state, reward, done = GAME.step(a) #does that step
        REPLAY_MEMORY.push((scale(state), a, reward, scale(new_state)))

        #experience replay is here
        if REPLAY_MEMORY.can_provide_sample(20):
            experiences = REPLAY_MEMORY.sample(20)
            target_vecs = []
            for j in range(len(experiences)):
                target = experiences[j][2] + y * np.max(model.predict(np.array([experiences[j][3]])))
                target_vec = model.predict(np.array([experiences[j][0]]))[0]
                target_vec[experiences[j][1]] = target
                target_vecs.append(target_vec)
            target_vecs = np.array(target_vecs)
            states = [s for s, _, _, _ in [exp for exp in experiences]]
            states = np.array(states)
            model.fit(states, target_vecs, epochs=1, verbose=1 if counter % 100 == 0 else 0)
        state = new_state
        if counter > 1200: #game runs for 20 seconds each episode
            done = True
            scores.append(GAME.PLAYER.score)
model.save("model.h5")

First, this takes a long time to train on my GTX1050. Is this normal for such a simple game? Also, does my code look fine? This is my first time with Deep Q Learning, so I would appreciate a second set of eyes.

What is happening is that training is super slow (more than an hour for 20 episodes (or 400 seconds of actual game play)). Also, it does not seem to get much better. The paddle (after 20 episodes) moves left and right but without any obvious pattern.

Here is a link to the code. Also, available on GitHub.

Answer 1

Your implementation of single-step Q-learning with neural network and experience replay is basically correct.

There are a few blocking issues preventing you seeing it working correctly.

Your main problem is a bug in your feature scaling routine. That is a Python issue, not really an AI one. In short, you scale the input features in-place multiple times, including an effective double-scaling of next_state (when it gets copied to state you scale it in place a second time in the next loop) so that all the states that you store in the experience replay table never match to any input states. You need to change your definition of scale to not do this. A very simple re-write of your routine would be:

def scale(s):
    return [s[0]/500, s[1]/500, s[2]/300, s[3]/3]

In addition, you need to change random action selection to:

np.random.randint(0, 3)

because the end of range is never output (this matches behaviour of other range values and operators in Python). Not including the "do nothing" action during exploration means that the agent will test it less, and have less data to work with to assess whether it is the best action. This is a minor issue for this environment, but you should fix it nonetheless.

What is happening is that training is super slow (more than an hour for 20 episodes (or 400 seconds of actual game play)).

I cannot replicate this fault and can train 20 episodes in around 2.5 minutes - that's over 20 times faster than you report. I am not using a GPU. Possibly in your case, Theano and Pygame are fighting for control of the GPU, or you may have a GPU configuration issue with Theano. Try turning GPU acceleraton off to verify whether it helps. You don't benefit much from a GPU for this environment (most time is spent in Python running the Q-learning and the environment), so can afford to put solving that issue to one side for now.

Also, it does not seem to get much better. The paddle (after 20 episodes) moves left and right but without any obvious pattern

Sadly, I cannot see the output at all on my MacBook pro, but I was able to use feedback of the expected score. A random agent gets a mean score of ~5.5 per episode. With the scale function corrected, and a rough guess at working hyperparameters, I can get an average score of ~17 per episode consistently after 60 episodes of training. After 150 episodes - taking 20 minutes to train - the agent was scoring ~20 per episode and I stopped there. It is possible that an expected score around 20 is already optimal, as it is a very simple environment, but I don't know.

Once you have a working system, there are lots of hyperparameters you could play with to try and improve this. I got my results by making the following changes after fixing the scale function:

• Starting epsilon of 1.0

• Repay memory size 10,000

• Only start learning when replay memory has greater than 1,000 entries

• Discount factor $\gamma$ 0.99

• Neural network with 20 neurons per layer with tanh activation instead of relu

There is quite a lot else you could change that might make the agent learn more effectively or perhaps aim for a more optimal policy. Have fun experimenting!