2
$\begingroup$

I am implementing a Deep Q-Learning Algorithm. The model appears to improve but after awhile it just crashes and does just as well as if an agent was making random decisions. Shouldn't the behavior get better and better as time goes on? I tried training 3 times and the same thing happened each time. Letting the model run longer never saw the model improve again and basically stayed random. I'm new to data science/machine learning and if someone could point me towards some resources to prevent this behavior I would very much appriciate it.

For some more context, I'm trying to solve the cart pole problem (inverted pendulum) using Q-Learning. The graph I've included below shows how long the model is able to balance the pendulum. The x-axis represents the episodes and the y-axis represents time balanced.

enter image description here

EDIT 1:

When the graph was generated, this is how the model was created. I never set a learning rate.

model = keras.models.Sequential()
model.add(tf.keras.layers.InputLayer(input_shape=(4,)))
model.add(tf.keras.layers.Dense(64))
model.add(tf.keras.layers.Dense(32))
model.add(tf.keras.layers.Dense(CartEnv.NUM_ACTIONS))
    
model.compile(loss='mse', optimizer='adam', metrics=['accuracy'])
return model

I've since changed it to this.

 model = keras.models.Sequential()
 model.add(tf.keras.layers.InputLayer(input_shape=(4,)))
 model.add(tf.keras.layers.Dense(64))
 model.add(tf.keras.layers.Dense(32))
 model.add(tf.keras.layers.Dense(CartEnv.NUM_ACTIONS))

opt = tf.keras.optimizers.Adam(LR)        # LR = 0.01
model.compile(loss='mse', optimizer=opt, metrics=['accuracy'])
return model

I have a 'target model' that gets updated every 5 'episodes.' An episode ends after 15s or when the angle is greater than 45 degrees. Since I've changed it to use the tf.keras.optimizers.Adam optimizer, the model progress looks very noisy. Since changing the optimizer, when I evaluate it while training, sometimes the agent will be very good then drop the to very bad the next time it is evaluated. (The agent is evaluated after training on 20 episodes. For an evaluation, the agent runs through 10 episodes and the time is averaged). Here is a plot of the training progress. I've tried changing the learning rate to be lower but the model doesn't appear to improve at all.

enter image description here

Would an evolutionary type algorithm be better because it takes the best of each generation? I feel like that wouldn't allow a model to drop like it does now. However, I would still like to know why what I'm doing now isn't working.

EDIT 2:

I've since changed things to use a learning rate of 0.00003 (3e-5) as suggested in the comments. I've also changed the target model to update every 1000 episodes. I'm still finidng similar results. The training progress graph is below (like before it has episodes on the x-axis and average time survived on the y-axis). I am training the model on every step of the episode, should I only be training after an episode completes?

Training EDIT3 The green vertical lines represent when the target model was updated. The red horizontal lines represent the min and max times. Ignore the x-axis. Model progress was recorded every 20 episodes so the x-axis should be scaled by 20.

EDIT 3:

I'm solving the cart pole problem. I want to reward behavior when the pole's angle is closer to upright (theta=0). I also want to reward behavior when the cart itself is closer to the center (x=0). I take away 150 from the reward when it tips over.

def get_reward(self):
    score = 0.0
    fail = False
    complete = False

    if abs(self.cart.theta) > CartEnv.THETA_MAX: # THETA_MAX = 45 degrees or pi/4 radians.
        fail = True
        complete = True
        score -= 150

    if not fail and self.time >= 15:
        complete = True

    # Take away score further from center
    score += 30 - (abs(self.cart.theta) * RAD_TO_DEG) # [30, -15]
    score += .5 * max(10 - abs(self.cart.x), -20) # [5, -10]

    return (score, complete)
$\endgroup$
13
  • $\begingroup$ Please, edit your post to provide more details about the hyper-parameters that you're using (e.g. learning rate, number of layers, etc.). Those might be useful to answer your question. $\endgroup$
    – nbro
    Mar 17 at 9:46
  • $\begingroup$ @nbro , I've added an edit. I'm new to the world of data science/machine learning so any advice would be appriciated! $\endgroup$ Mar 18 at 1:58
  • $\begingroup$ I would choose a learning rate way smaller. Most of implementations that I have seen assume something around 3e-5. Also, the rate that you're updating the target net seems too short. DQN are know to be unstable and the idea of use a target net aims circumvent it. Perhaps try to use something above ~1000 and let me know if your agent improved. $\endgroup$
    – HenDoNR
    Mar 18 at 9:04
  • $\begingroup$ @HenDoNR, under 'EDIT 2' you can see the results from the changes you suggested. The model appears to make changes after the target model is updated. Should I update the target model only when the model appears to be doing well? $\endgroup$ Mar 23 at 12:11
  • $\begingroup$ Thanks to give more details. You definitely shouldn't train your agent after the end of each episode. The idea of Experience Replay/Replay memory is store the experiences and shuffled them to break the correlation between data. It's always hard to put parameters in numbers, but try to increase substantially the amount of experiences stored before train your agent. I would say the same for you target net update rate $\endgroup$
    – HenDoNR
    Mar 24 at 13:17

1 Answer 1

1
$\begingroup$

You report that your model is configured as follows:

model = keras.models.Sequential()
model.add(tf.keras.layers.InputLayer(input_shape=(4,)))
model.add(tf.keras.layers.Dense(64))
model.add(tf.keras.layers.Dense(32))
model.add(tf.keras.layers.Dense(CartEnv.NUM_ACTIONS))

opt = tf.keras.optimizers.Adam(LR)        # LR = 0.01
model.compile(loss='mse', optimizer=opt, metrics=['accuracy'])
return model

It is important to note that the TensorFlow/Keras documentation reports regarding the activation keyword parameter:

[The activation parameter allows you to specify the] Activation function to use. If you don't specify anything, no activation is applied (ie. "linear" activation: a(x) = x).

Currently, your network is using the default of None, which is equivalent to linear. Having multiple layers but only using a linear activation is akin to performing multiple linear transformations in series; the entire series is always collapsible to a single transformation matrix.

In other words, since all of your layers are linear, the network does not really gain complexity by adding additional layers. In effect you have only one hidden layer.

You might try simply adding a non-linear activation. Perhaps:

model = keras.models.Sequential()
model.add(tf.keras.layers.InputLayer(input_shape=(4,)))
model.add(tf.keras.layers.Dense(64, activation='relu'))
model.add(tf.keras.layers.Dense(32, activation='relu'))
model.add(tf.keras.layers.Dense(CartEnv.NUM_ACTIONS))

opt = tf.keras.optimizers.Adam(LR)        # LR = 0.01
model.compile(loss='mse', optimizer=opt, metrics=['accuracy'])
return model
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .