0
$\begingroup$

I am trying to write a DQN model that will be able to solve OpenAI gym CartPole environment. I successfully managed to do it using scalar observation data that env.step() returns. But I wanted to make a DQN that would learn from pixels so I made images returned by env.render(mode='rgb_array') as my states. Unfortunately I could not get it to work.

I am stacking the frames to capture the sense of motion (n_frames is equal to 3 here)

def stack_frames(frame, is_new):
    ''' stacks n_frames amount of frames to generate single sample of input,
        used to capture sense of motion '''
    global stacked_frames
    frame = preprocess(frame)
    if is_new:
        stacked_frames = deque([np.zeros((frame.shape[0], frame.shape[1]), dtype=np.int32) for i in range(n_frames)], 
                       maxlen=n_frames)
        # for new episodes first frame is appended n_frames times
        for i in range(n_frames):
            stacked_frames.append(frame)
    else:
        # otherwise frame goes into buffer
        stacked_frames.append(frame)

    return np.stack(stacked_frames, axis=-1)

preprocessing the frames

def preprocess(frame):
    ''' crops the image, converts to grayscale'''
    frame = rgb2gray(frame) # greyscale frame 
    frame = frame[25:, :] # crop
    frame = transform.resize(frame, [80, 80])

    return frame

and added convolutional layers to my model

def dqn(input_shape):
    In = Input(shape=input_shape)
    x = Conv2D(filters=16, kernel_size=(8,8), strides=4, padding='same', activation='relu')(In)
    x = Conv2D(filters=32, kernel_size=(4,4), strides=2, padding='same', activation='relu')(x)
    x = Flatten()(x)
    x = Dense(64, activation='relu')(x)
    x = Dense(16, activation='relu')(x)
    Out = Dense(env.action_space.n)(x)

    model = Model(In, Out)
    model.compile(loss='mse', optimizer=RMSprop(learning_rate=alpha))

    return model

I have left everything else unchanged, but model still does not learn. After 1000 episodes (where each episode lasts until agent looses) the total rewards have still not improved.

What can I do to fix my model? I know DQN are super sensitive to changes in hyperparameter values, but I am not sure what to change. I am using learning rate of 0.001 and discount factor of 0.99. I have tried using huber loss as well, but it did not improve anything. Any help would be appreciated.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.