Deep Q-Learning poor convergence on Stochastic Environment

Question

I'm trying to implement a Deep Q-network in Keras/TF that learns to play Minesweeper (our stochastic environment). I have noticed that the agent learns to play the game pretty well with both small and large board sizes. However, it only converges/learns when the layout of the mines is the same for each game. That is, if I randomize the mine distribution from game to game, the agent learns nothing - or near to it. I tried using various network architectures and hyperparameters but to no avail.

I tried a lot of network architectures including:

1. The input to the network is the entire board matrix, with the individual cells having values of -1 if unrevealed, or 0 to 8 if revealed.
2. The output of the network is also the entire board representing the desirability of clicking each cell.
3. Tried fully connected hidden layers (both wide and deep).
4. Tried Convolutional hidden layers (tried stacked them, using different kernel sizes, padding, etc.).
5. Tried adding Dropout after hidden layers too.

Is DQN applicable for environments that change every episode or have I approached this from the wrong side?

It seems no matter the network architecture, the agent won't learn. Any input is greatly appreciated. Please let me know if you require any code or further explanations.

Answer 1

The inputs that you describe seem like they should be sufficient for a DQN-based agent to learn a good strategy for playing Minesweeper, regardless of whether or not the starting layout changes. The inputs contain all information that is necessary.

However, the problem certainly becomes much easier (probably too easy) if the initial problem is always the same. The DQN algorithm will be very likely to "pick up" on this trend and "exploit" it. The inputs may be sufficient to learn a more general Minesweeper strategy, but if it is consistently faced with the same "level" every single time, it will be much easier for the DQN algorithm to just memorize exactly where the mines are and play perfectly based on that memory, rather than any actual strategy. Due to the way learning is implemented (based on gradient descent), the algorithm will generally tend to converge to such an easily-reachable "memorization" strategy rather than something that is actually "smart".

For that reason, I do think training will be much slower in the case where the layout is randomized. I'm not just thinking along the lines of e.g. twice as slow here, but would expect multiple orders of magnitude more experience required for successful learning. That's just my educated guess though, never tried training DQN for minesweeper specifically so can't tell for sure. It might also require more elaborate hyperparameter tuning, and maybe require a different network architecture (probably require a larger network).