# What is the optimal exploration-exploitation trade-off in Q*bert?

I am training an RL agent with Deep Q-learning + Experience Replay on the Q*bert Atari environment. After 400,000 frames, my agent appears to have learned strategic information about the game, but none about the environment. It has learned that a good immediate strategy is to simply jump down both diagonals and fall of the board, thus completing a large portion of the first level. However, it remains to understand neither the boundaries of the board to prevent jumping off, nor anything about avoiding enemies. I’m asking this here, instead of Stack Overflow because it is a more general question with less of a need in terms of programming understanding. Simply, I am asking whether or not this is a matter of a pore exploration policy (which I presume). If you agree, what should be a better exploration policy for Q*bert that would facilitate my agent’s learning experience?

As per the request of a comment:

Could you add what your current exploration approach is, and what options you are using for your Deep Q Learning implementation (e.g. replay size, batch size, NN architecture, steps per target network copy, or if you are using a different update mechanism for the target network). Also if you are using any other approach different to the classic DQN paper such as in state representation.

Here are my parameters:

• Exploration policy: epsilon = min(1.0, 1000 / (frames + 1))
• Replay Memory = 20,000 frames
• Batch size = 32 transitions
• NN architecture: Conv2D(64, 3, 2), Dropout(0.2), Dense(32, relu), Dense(32, relu), Dense(num_actions, linear)
• Steps per target network copy: 100
• Could you add what your current exploration approach is, and what options you are using for your Deep Q Learning implementation (e.g. replay size, batch size, NN architecture, steps per target network copy, or if you are using a different update mechanism for the target network). Also if you are using any other approach different to the classic DQN paper such as in state representation. Aug 27, 2020 at 20:24
• I’ve added the information you requested. Aug 27, 2020 at 20:43

I can spot three, maybe four, things in your implementation that could be contributing to incomplete learning that you are observing.

## More exploration in long term

I think you have correctly identified that exploration could be an issue. In off-policy learning (which Q-learning is an instance of), it is usual to set a minimum exploration rate. It is a hyperparameter that you need to manage. Set too high, the agent will never experience the best rewards as it will make too many mistakes. Set too low, the agent will not explore enough to find the correct alternative actions when the opportunity to learn them occurs.

I would suggest for you something like:

epsilon = max(min(1.0, 1000 / (frames + 1)), 0.01)


You can choose numbers other than 0.01, but I think that is a reasonable start for many Atari games. You could try higher, up to 0.1 in games which are more forgiving of mistakes.

## Remove dropout

I am not sure why, but I always have problems with dropout in RL neural networks. Try removing the dropout layer.

## More convolutional layers

Convolutional layers are very efficient generalisers for vision and grid-based problems. You won't really benefit much from having a single layer though. I would add another two, increase the number of output channels.

## Maybe state representation?

It is not clear from your description whether you are using a single colour frame for the state representation, or stacked greyscale frames for the last 3 inputs. It should be the latter, and if you want to more closely replicate the orginal DQN Atari paper, you should take the previous 4 frames as input.

In addition, you should be normalising the input into range $$[0,1]$$ or $$[-1,1]$$. The native image range $$[0,255]$$ is tricky for neural networks to process, and quite common for value functions to get stuck if you don't normalise.

• This really helped, thanks! The reward graph looks much more normal now. It more greatly resembles what I see in the paper and related works. Aug 28, 2020 at 15:26
• @RyanRudes: No problem. Which parts out of interest did you need to change? Aug 28, 2020 at 15:37
• I also just realized that the paper had trained the agent for 5 million frames for most of their graphs. Looking at the charts in the paper, reward didn’t even rise significantly until 500,000 frames in, and I hadn’t even trained up to there yet. So I severely underestimated the amount of training these require. Aug 28, 2020 at 15:39
• I took into account all of your suggestions (I already had the normalization implemented though). They were very helpful, thanks. Aug 28, 2020 at 15:48