I am trying to write a DQN model that will be able to solve OpenAI gym CartPole environment. I successfully managed to do it using scalar observation data that env.step() returns. But I wanted to make a DQN that would learn from pixels so I made images returned by env.render(mode='rgb_array') as my states. Unfortunately I could not get it to work.

I am stacking the frames to capture the sense of motion (n_frames is equal to 3 here)

def stack_frames(frame, is_new):
    ''' stacks n_frames amount of frames to generate single sample of input,
        used to capture sense of motion '''
    global stacked_frames
    frame = preprocess(frame)
    if is_new:
        stacked_frames = deque([np.zeros((frame.shape[0], frame.shape[1]), dtype=np.int32) for i in range(n_frames)], 
        # for new episodes first frame is appended n_frames times
        for i in range(n_frames):
        # otherwise frame goes into buffer

    return np.stack(stacked_frames, axis=-1)

preprocessing the frames

def preprocess(frame):
    ''' crops the image, converts to grayscale'''
    frame = rgb2gray(frame) # greyscale frame 
    frame = frame[25:, :] # crop
    frame = transform.resize(frame, [80, 80])

    return frame

and added convolutional layers to my model

def dqn(input_shape):
    In = Input(shape=input_shape)
    x = Conv2D(filters=16, kernel_size=(8,8), strides=4, padding='same', activation='relu')(In)
    x = Conv2D(filters=32, kernel_size=(4,4), strides=2, padding='same', activation='relu')(x)
    x = Flatten()(x)
    x = Dense(64, activation='relu')(x)
    x = Dense(16, activation='relu')(x)
    Out = Dense(env.action_space.n)(x)

    model = Model(In, Out)
    model.compile(loss='mse', optimizer=RMSprop(learning_rate=alpha))

    return model

I have left everything else unchanged, but model still does not learn. After 1000 episodes (where each episode lasts until agent looses) the total rewards have still not improved.

What can I do to fix my model? I know DQN are super sensitive to changes in hyperparameter values, but I am not sure what to change. I am using learning rate of 0.001 and discount factor of 0.99. I have tried using huber loss as well, but it did not improve anything. Any help would be appreciated.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.