I am training an RL agent with Deep Q-learning + Experience Replay on the Q*bert Atari environment. After 400,000 frames, my agent appears to have learned strategic information about the game, but none about the environment. It has learned that a good immediate strategy is to simply jump down both diagonals and fall of the board, thus completing a large portion of the first level. However, it remains to understand neither the boundaries of the board to prevent jumping off, nor anything about avoiding enemies. I’m asking this here, instead of Stack Overflow because it is a more general question with less of a need in terms of programming understanding. Simply, I am asking whether or not this is a matter of a pore exploration policy (which I presume). If you agree, what should be a better exploration policy for Q*bert that would facilitate my agent’s learning experience?
As per the request of a comment:
Could you add what your current exploration approach is, and what options you are using for your Deep Q Learning implementation (e.g. replay size, batch size, NN architecture, steps per target network copy, or if you are using a different update mechanism for the target network). Also if you are using any other approach different to the classic DQN paper such as in state representation.
Here are my parameters:
- Exploration policy: epsilon =
min(1.0, 1000 / (frames + 1))
- Replay Memory = 20,000 frames
- Batch size = 32 transitions
- NN architecture: Conv2D(64, 3, 2), Dropout(0.2), Dense(32, relu), Dense(32, relu), Dense(num_actions, linear)
- Steps per target network copy: 100