I am trying to write a DQN model that will be able to solve OpenAI gym CartPole environment. I successfully managed to do it using scalar observation data that
env.step() returns. But I wanted to make a DQN that would learn from pixels so I made images returned by
env.render(mode='rgb_array') as my states. Unfortunately I could not get it to work.
I am stacking the frames to capture the sense of motion (n_frames is equal to 3 here)
def stack_frames(frame, is_new): ''' stacks n_frames amount of frames to generate single sample of input, used to capture sense of motion ''' global stacked_frames frame = preprocess(frame) if is_new: stacked_frames = deque([np.zeros((frame.shape, frame.shape), dtype=np.int32) for i in range(n_frames)], maxlen=n_frames) # for new episodes first frame is appended n_frames times for i in range(n_frames): stacked_frames.append(frame) else: # otherwise frame goes into buffer stacked_frames.append(frame) return np.stack(stacked_frames, axis=-1)
preprocessing the frames
def preprocess(frame): ''' crops the image, converts to grayscale''' frame = rgb2gray(frame) # greyscale frame frame = frame[25:, :] # crop frame = transform.resize(frame, [80, 80]) return frame
and added convolutional layers to my model
def dqn(input_shape): In = Input(shape=input_shape) x = Conv2D(filters=16, kernel_size=(8,8), strides=4, padding='same', activation='relu')(In) x = Conv2D(filters=32, kernel_size=(4,4), strides=2, padding='same', activation='relu')(x) x = Flatten()(x) x = Dense(64, activation='relu')(x) x = Dense(16, activation='relu')(x) Out = Dense(env.action_space.n)(x) model = Model(In, Out) model.compile(loss='mse', optimizer=RMSprop(learning_rate=alpha)) return model
I have left everything else unchanged, but model still does not learn. After 1000 episodes (where each episode lasts until agent looses) the total rewards have still not improved.
What can I do to fix my model? I know DQN are super sensitive to changes in hyperparameter values, but I am not sure what to change. I am using learning rate of 0.001 and discount factor of 0.99. I have tried using huber loss as well, but it did not improve anything. Any help would be appreciated.