What are disadvantages/limitations of Monte Carlo Tree Search in RL?

Question

What are disadvantages/limitations of Monte Carlo Tree Search in RL, and hence for what kind of applications might its use not be appropriate?

Answer 1

The hard to work around traits of Monte Carlo Tree Search (MCTS) are:

• It is model-based, and due to using a long look-ahead distance it can be sensitive to errors in the model (for learned models).

• It requires compute resources at the time of making a decision. In some domains this is an advantage (the compute resource is efficiently applied to a required decision), in others it is a liability where it may be hard to see much benefit on a real time system.

The first issue is strongly limiting - you need a good predictive model of the environment, and this is often not available. Even simple physical systems are hard to model accurately a significant amount into the future.

The second issue might be worked around by e.g. only using MCTS as a training assistant to develop a policy function - as MCTS evaluates and chooses an action more accurately than the current value and policy functions, it can be useful as updates to those functions (this is basically how AlphaGo training works). You would expect that policy function to perform worse without MCTS for look ahead planning than during training, but it could still be an effective agent.