1
$\begingroup$

This is a follow-up question about one I asked earlier. The first question is here. Basically, I have a game where a paddle moves left and right to catch as much "food" as possible. Some food is good (gain points) and some is bad (lose points). NN Architecture:

    #inputs - paddle.x, food.x, food.y, food.type
    #moves: left, right, stay
    model = Sequential()
    model.add(Dense(10, input_shape=(4,), activation='relu'))
    model.add(Dense(10, activation='relu'))
    model.add(Dense(3, activation='linear'))
    model.compile(loss='mean_squared_error', optimizer='adam')

As suggested in the other question, I scaled my inputs to be between 0 and 1. Also, implemented experience replay (although I am not confident I did it correctly).

Here is my ReplayMemory class:

class ReplayMemory():
    def __init__(self, capacity):
        self.capacity = capacity
        self.memory = []
        self.count = 0

    def push(self, experience):
        if len(self.memory) < self.capacity:
            self.memory.append(experience)
        else:
            self.memory[self.count % self.capacity] = experience
        self.count += 1

    def sample(self, batch_size):
        return random.sample(self.memory, batch_size)

    def can_provide_sample(self, batch_size):
        return len(self.memory) >= batch_size

This basically stores states/rewards/actions and returns a random group when asked.

Lastly, here is my learning code:

def learning(num_episodes=20):
        global scores, experiences, target_vecs
        y = 0.8
        eps = 0
        decay_factor = 0.9999

for i in range(num_episodes):
    state = GAME.reset()
    GAME.done = False
    done = False
    counter = 0
    while not done:
        eps *= decay_factor
        counter+=1

        if np.random.random() < eps:
            a = np.random.randint(0, 2)
        else:
            a = np.argmax(model.predict(np.array([scale(state)])))

        new_state, reward, done = GAME.step(a) #does that step
        REPLAY_MEMORY.push((scale(state), a, reward, scale(new_state)))

        #experience replay is here
        if REPLAY_MEMORY.can_provide_sample(20):
            experiences = REPLAY_MEMORY.sample(20)
            target_vecs = []
            for j in range(len(experiences)):
                target = experiences[j][2] + y * np.max(model.predict(np.array([experiences[j][3]])))
                target_vec = model.predict(np.array([experiences[j][0]]))[0]
                target_vec[experiences[j][1]] = target
                target_vecs.append(target_vec)
            target_vecs = np.array(target_vecs)
            states = [s for s, _, _, _ in [exp for exp in experiences]]
            states = np.array(states)
            model.fit(states, target_vecs, epochs=1, verbose=1 if counter % 100 == 0 else 0)
        state = new_state
        if counter > 1200: #game runs for 20 seconds each episode
            done = True
            scores.append(GAME.PLAYER.score)
model.save("model.h5")

First, this takes a long time to train on my GTX1050. Is this normal for such a simple game? Also, does my code look fine? This is my first time with Deep Q Learning, so I would appreciate a second set of eyes.

What is happening is that training is super slow (more than an hour for 20 episodes (or 400 seconds of actual game play)). Also, it does not seem to get much better. The paddle (after 20 episodes) moves left and right but without any obvious pattern.

Here is a link to the code. Also, available on GitHub.

$\endgroup$
  • $\begingroup$ @NeilSlater Yes. Would you like me to just post the code or is there a better way to share it? $\endgroup$ – Nick Solonko Aug 6 at 20:14
  • $\begingroup$ @NeilSlater Ok, I added a link. Thanks! $\endgroup$ – Nick Solonko Aug 6 at 21:23
  • $\begingroup$ To run the game using the model, I first run model.save("model.h5").. The run the game. I create a game object g=Game(). Then, I run g.run(load_model("model.h5")). Let me know if this helps @NeilSlater $\endgroup$ – Nick Solonko Aug 8 at 12:59
  • $\begingroup$ @NeilSlater Also, when training, I don't actually run the game visually. Just using the step function that does all the math behind it but not the drawing. I also fixed an error where the inputs were not scaled when making predictions (when testing mode). The new code is on Github. $\endgroup$ – Nick Solonko Aug 8 at 21:41
1
$\begingroup$

Your implementation of single-step Q-learning with neural network and experience replay is basically correct.

There are a few blocking issues preventing you seeing it working correctly.

Your main problem is a bug in your feature scaling routine. That is a Python issue, not really an AI one. In short, you scale the input features in-place multiple times, including an effective double-scaling of next_state (when it gets copied to state you scale it in place a second time in the next loop) so that all the states that you store in the experience replay table never match to any input states. You need to change your definition of scale to not do this. A very simple re-write of your routine would be:

def scale(s):
    return [s[0]/500, s[1]/500, s[2]/300, s[3]/3]

In addition, you need to change random action selection to:

np.random.randint(0, 3)

because the end of range is never output (this matches behaviour of other range values and operators in Python). Not including the "do nothing" action during exploration means that the agent will test it less, and have less data to work with to assess whether it is the best action. This is a minor issue for this environment, but you should fix it nonetheless.

What is happening is that training is super slow (more than an hour for 20 episodes (or 400 seconds of actual game play)).

I cannot replicate this fault and can train 20 episodes in around 2.5 minutes - that's over 20 times faster than you report. I am not using a GPU. Possibly in your case, Theano and Pygame are fighting for control of the GPU, or you may have a GPU configuration issue with Theano. Try turning GPU acceleraton off to verify whether it helps. You don't benefit much from a GPU for this environment (most time is spent in Python running the Q-learning and the environment), so can afford to put solving that issue to one side for now.

Also, it does not seem to get much better. The paddle (after 20 episodes) moves left and right but without any obvious pattern

Sadly, I cannot see the output at all on my MacBook pro, but I was able to use feedback of the expected score. A random agent gets a mean score of ~5.5 per episode. With the scale function corrected, and a rough guess at working hyperparameters, I can get an average score of ~17 per episode consistently after 60 episodes of training. After 150 episodes - taking 20 minutes to train - the agent was scoring ~20 per episode and I stopped there. It is possible that an expected score around 20 is already optimal, as it is a very simple environment, but I don't know.

Once you have a working system, there are lots of hyperparameters you could play with to try and improve this. I got my results by making the following changes after fixing the scale function:

  • Starting epsilon of 1.0

  • Repay memory size 10,000

  • Only start learning when replay memory has greater than 1,000 entries

  • Discount factor $\gamma$ 0.99

  • Neural network with 20 neurons per layer with tanh activation instead of relu

There is quite a lot else you could change that might make the agent learn more effectively or perhaps aim for a more optimal policy. Have fun experimenting!

$\endgroup$
  • $\begingroup$ Thanks for taking the time to try my code! I implemented all that you said. I tried using tensorflow instead of tensorflow-gpu in order to just use the CPU. It is still just as slow. This loop: for j in range(len(experiences)): takes a lot of time I guess. Could this be an IDE issue? I am using Spyder. I also completely removed PyGame from the code (just while training). $\endgroup$ – Nick Solonko Aug 14 at 21:33
  • $\begingroup$ @NickSolonko: Sorry I cannot tell why you are experiencing such slow times. All I can say is that is not due to your code in general which runs fine on my MacBook (which is 2012 model, so not top of the range for a modern machine). You could vectorise that loop in Keras - if you are not sure how to, ask here or on SO, or on DataScience. Not sure if that will help you that much though, as the core problem seems to be somewhere else. $\endgroup$ – Neil Slater Aug 15 at 8:15
  • $\begingroup$ Did it take your system 2.5 minutes even when sampling 1,000 Replay Memory records? I tried another Python IDE on another computer. It is a little faster, but will ask another question on Stack to try to fix the problem. $\endgroup$ – Nick Solonko Aug 17 at 13:25
  • 1
    $\begingroup$ @NickSolonko: I didn't try with just 1000, and cannot at the moment. I upgraded it to 10,000 for my test. I doubt my change would make it faster, as the 2.5 minutes was for 20 episodes (and 20 minutes total to reach top performance for the agent), and sampling from 10,000 entries would be slightly slower in Python, probably not noticeable though $\endgroup$ – Neil Slater Aug 17 at 15:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.