This is a follow-up question about one I asked earlier. The first question is here. Basically, I have a game where a paddle moves left and right to catch as much "food" as possible. Some food is good (gain points) and some is bad (lose points). NN Architecture:
#inputs - paddle.x, food.x, food.y, food.type #moves: left, right, stay model = Sequential() model.add(Dense(10, input_shape=(4,), activation='relu')) model.add(Dense(10, activation='relu')) model.add(Dense(3, activation='linear')) model.compile(loss='mean_squared_error', optimizer='adam')
As suggested in the other question, I scaled my inputs to be between 0 and 1. Also, implemented experience replay (although I am not confident I did it correctly).
Here is my ReplayMemory class:
class ReplayMemory(): def __init__(self, capacity): self.capacity = capacity self.memory =  self.count = 0 def push(self, experience): if len(self.memory) < self.capacity: self.memory.append(experience) else: self.memory[self.count % self.capacity] = experience self.count += 1 def sample(self, batch_size): return random.sample(self.memory, batch_size) def can_provide_sample(self, batch_size): return len(self.memory) >= batch_size
This basically stores states/rewards/actions and returns a random group when asked.
Lastly, here is my learning code:
def learning(num_episodes=20): global scores, experiences, target_vecs y = 0.8 eps = 0 decay_factor = 0.9999 for i in range(num_episodes): state = GAME.reset() GAME.done = False done = False counter = 0 while not done: eps *= decay_factor counter+=1 if np.random.random() < eps: a = np.random.randint(0, 2) else: a = np.argmax(model.predict(np.array([scale(state)]))) new_state, reward, done = GAME.step(a) #does that step REPLAY_MEMORY.push((scale(state), a, reward, scale(new_state))) #experience replay is here if REPLAY_MEMORY.can_provide_sample(20): experiences = REPLAY_MEMORY.sample(20) target_vecs =  for j in range(len(experiences)): target = experiences[j] + y * np.max(model.predict(np.array([experiences[j]]))) target_vec = model.predict(np.array([experiences[j]])) target_vec[experiences[j]] = target target_vecs.append(target_vec) target_vecs = np.array(target_vecs) states = [s for s, _, _, _ in [exp for exp in experiences]] states = np.array(states) model.fit(states, target_vecs, epochs=1, verbose=1 if counter % 100 == 0 else 0) state = new_state if counter > 1200: #game runs for 20 seconds each episode done = True scores.append(GAME.PLAYER.score) model.save("model.h5")
First, this takes a long time to train on my GTX1050. Is this normal for such a simple game? Also, does my code look fine? This is my first time with Deep Q Learning, so I would appreciate a second set of eyes.
What is happening is that training is super slow (more than an hour for 20 episodes (or 400 seconds of actual game play)). Also, it does not seem to get much better. The paddle (after 20 episodes) moves left and right but without any obvious pattern.