DQN and MCTS (+ NNs) are algorithms of two distinct RL flavors:
- DQN is model-free RL: it's able to learn the value of actions without having access to the underlying MDP (i.e. environment's model), so that you can later retrieve a ($\epsilon$-greedy) policy.
- MCTS is a planning technique in model-based RL. You can use it only if you have access to the model of the MDP (or to a learned approximation). Having the true model is quite beneficial, because you can plan in advance meaning that you can reason about which action is the best by seeing what would happen in the future.
DQN is able to do a 1-step lookahead in the future by looking at the action-value of the next state-action pair, i.e. $Q(s',a')$ - there are also extensions to $n$-step learning, anyway $n$ is relatively small (because that whould increase variance a lot) - instead MCTS can explore the full tree of interaction (where you have all the possible state-action combinations) until termination of the episode. Indeed, exploring the full tree is always impractical and MCTS does this in a randomized fashion (thus the name Monte Carlo) to approximate the value.
AlphaGo/Zero uses also a value network to provide a further estimate of the next state value, $V(s')$, which is combined with the classical notion of value provided by MCTS itself, then it has also a policy network that is learned to predict the actions found useful by MCTS.
My current theory is that Deep Q Learning approaches require dense reward environments and games like chess don't really place emphasis on mid game rewards.
Partially true, I mean exploration technique and reward shaping can help with sparse rewards, but the point is that having access to the underlying model makes everything easier to learn. Consider that DQN has to implicitly learn the model (actually it learns an association between state-action and value), and after learning it cannot change its beliefs if the state distribution shifts for example. Instead, MCTS can even adapt to changing distributions because you plan your action in advance, so without experiencing the actual environment.
My question is why didn't the Alpha zero team use a deep Q learning approach if it is capable of building complex representations of the state.
Their value network can, in principle, learn complex representations of the state too. It is used as a heuristic to improve the search of MCTS. In principle, I think is possibile to use a Q-network too but one has to change the heuristic that MCTS follows to judge actions which guides the exploration of the game tree.
