I am implementing a Deep Q-Learning Algorithm. The model appears to improve but after awhile it just crashes and does just as well as if an agent was making random decisions. Shouldn't the behavior get better and better as time goes on? I tried training 3 times and the same thing happened each time. Letting the model run longer never saw the model improve again and basically stayed random. I'm new to data science/machine learning and if someone could point me towards some resources to prevent this behavior I would very much appriciate it.
For some more context, I'm trying to solve the cart pole problem (inverted pendulum) using Q-Learning. The graph I've included below shows how long the model is able to balance the pendulum. The x-axis represents the episodes and the y-axis represents time balanced.
When the graph was generated, this is how the model was created. I never set a learning rate.
model = keras.models.Sequential() model.add(tf.keras.layers.InputLayer(input_shape=(4,))) model.add(tf.keras.layers.Dense(64)) model.add(tf.keras.layers.Dense(32)) model.add(tf.keras.layers.Dense(CartEnv.NUM_ACTIONS)) model.compile(loss='mse', optimizer='adam', metrics=['accuracy']) return model
I've since changed it to this.
model = keras.models.Sequential() model.add(tf.keras.layers.InputLayer(input_shape=(4,))) model.add(tf.keras.layers.Dense(64)) model.add(tf.keras.layers.Dense(32)) model.add(tf.keras.layers.Dense(CartEnv.NUM_ACTIONS)) opt = tf.keras.optimizers.Adam(LR) # LR = 0.01 model.compile(loss='mse', optimizer=opt, metrics=['accuracy']) return model
I have a 'target model' that gets updated every 5 'episodes.' An episode ends after 15s or when the angle is greater than 45 degrees.
Since I've changed it to use the
tf.keras.optimizers.Adam optimizer, the model progress looks very noisy. Since changing the optimizer, when I evaluate it while training, sometimes the agent will be very good then drop the to very bad the next time it is evaluated. (The agent is evaluated after training on 20 episodes. For an evaluation, the agent runs through 10 episodes and the time is averaged). Here is a plot of the training progress. I've tried changing the learning rate to be lower but the model doesn't appear to improve at all.
Would an evolutionary type algorithm be better because it takes the best of each generation? I feel like that wouldn't allow a model to drop like it does now. However, I would still like to know why what I'm doing now isn't working.
I've since changed things to use a learning rate of 0.00003 (3e-5) as suggested in the comments. I've also changed the target model to update every 1000 episodes. I'm still finidng similar results. The training progress graph is below (like before it has episodes on the x-axis and average time survived on the y-axis). I am training the model on every step of the episode, should I only be training after an episode completes?
The green vertical lines represent when the target model was updated. The red horizontal lines represent the min and max times. Ignore the x-axis. Model progress was recorded every 20 episodes so the x-axis should be scaled by 20.
I'm solving the cart pole problem. I want to reward behavior when the pole's angle is closer to upright (theta=0). I also want to reward behavior when the cart itself is closer to the center (x=0). I take away 150 from the reward when it tips over.
def get_reward(self): score = 0.0 fail = False complete = False if abs(self.cart.theta) > CartEnv.THETA_MAX: # THETA_MAX = 45 degrees or pi/4 radians. fail = True complete = True score -= 150 if not fail and self.time >= 15: complete = True # Take away score further from center score += 30 - (abs(self.cart.theta) * RAD_TO_DEG) # [30, -15] score += .5 * max(10 - abs(self.cart.x), -20) # [5, -10] return (score, complete)