How to use convolution neural network in Deep-Q?

Question

I currently have a grid of pixels 20x20. Each pixel can be red green blue or black. So I have one hot-encoded the pixels giving a 20x20x4 array for each screen.

For my Deep-Q Network, I have attached two successive screenshots of the screen together giving a 20x20x4x2 array.

I am trying to build a Convolutional Neural Network to estimate the Q values but I am not sure if my current architecture is a good idea. It currently is as shown below:

    def create_model(self):
        model = Sequential()
        model.add(Conv3D(256, (4, 4,2), input_shape=(20,20,4,2)))
        model.add(Activation('relu'))
        model.add(Dropout(0.2))

        model.add(Conv3D(256, (2,2,1), input_shape=self.input_shape))
        model.add(Activation('relu'))

        model.add(Flatten())
        model.add(Dense(64))
        model.add(Dense(self.num_actions, activation='linear'))
        model.compile(loss='mse', optimizer=Adam(self.learning_rate), metrics=['accuracy'])
        return model

Is a 3d convolution a good idea? Is 256 filters a good idea? Are the filters (4,4,2) and (2,2,1) suitable? I realise answers may be highly subjective but I'm just looking for someone to point out any immediate flaws in the architecture.

Answer 1

Is a 3d convolution a good idea? Is 256 filters a good idea? Are the filters (4,4,2) and (2,2,1) suitable?

It's not so much that answers are subjective, but you are performing an experiment, and this should be driven by results. If you can find something published about a similar environment that might help you narrow down your choices.

That said, intuitively you don't gain much going from 2d to 3d convolutions when one of your dimensions is only sized 2. Purely from gut feeling I would suggest simply concatenating your two frames by having 8 channels instead of 4, and use 2d filters. This is simple to enough try and compare though, so perhaps you can do both.

You will likely want to explore deeper networks that have fewer initial filters and build up over a few more than two convolutional layers.

Definitely try your network without the dropout layer. I have not had much luck with using dropout in DQN, and I have seen others having similar problems. I am not sure what the exact issue is.

The accuracy metric won't do you much good on a regression task, so you can drop that and just use MSE loss. Bear in mind that training loss is a less useful metric overall in RL, since the prediction target (action value) is continuously changing as the policy changes. Low loss values mean that the value predictions are self-consistent, they don't necessarily mean that the learning has converged to an optimal policy.