Skip to main content

Questions tagged [monte-carlo-tree-search]

For questions related to Monte Carlo Tree Search (MCTS), which is a best-first, rollout-based tree search algorithm. MCTS gradually improves its evaluations of nodes in the trees using (semi-)random rollouts through those nodes, focusing a larger proportion of rollouts on the parts of the tree that are the most promising.

Filter by
Sorted by
Tagged with
19 votes
3 answers
11k views

How do I choose the best algorithm for a board game like checkers?

How do I choose the best algorithm for a board game like checkers? So far, I have considered only three algorithms, namely, minimax, alpha-beta pruning, and Monte Carlo tree search (MCTS). Apparently,...
JoeyB's user avatar
  • 477
17 votes
1 answer
3k views

How does "Monte-Carlo search" work?

I have heard about this concept in a Reddit post about AlphaGo. I have tried to go through the paper and the article, but could not really make sense of the algorithm. So, can someone give an easy-to-...
Dawny33's user avatar
  • 1,371
15 votes
3 answers
4k views

Does Monte Carlo tree search qualify as machine learning?

To the best of my understanding, the Monte Carlo tree search (MCTS) algorithm is an alternative to minimax for searching a tree of nodes. It works by choosing a move (generally, the one with the ...
Inertial Ignorance's user avatar
14 votes
3 answers
4k views

MCTS for non-deterministic games with very high branching factor for chance nodes

I'm trying to use a Monte Carlo Tree Search for a non-deterministic game. Apparently, one of the standard approaches is to model non-determinism using chance nodes. The problem for this game is that ...
Mark's user avatar
  • 241
11 votes
1 answer
785 views

Monte Carlo Tree Search: What kind of moves can easily be found and what kinds make trouble?

I want to start with a scenario that got me thinking about how well MCTS can perform: Let's assume there is a move that is not yet added to the search tree. It is some layers/moves too deep. But if we ...
Nocta's user avatar
  • 111
10 votes
3 answers
2k views

Why does Monte Carlo work when a real opponent's behavior may not be random

I am learning about Monte Carlo algorithms and struggling to understand the following: If simulations are based on random moves, how can the modeling of the opponent's behavior work well? For ...
kgautron's user avatar
  • 211
10 votes
1 answer
8k views

When should Monte Carlo Tree search be chosen over MiniMax?

I would like to ask whether MCTS is usually chosen when the branching factor for the states that we have available is large and not suitable for Minimax. Also, other than MCTS simluates actions, where ...
R AND B's user avatar
  • 101
8 votes
2 answers
1k views

What is the appropriate way to deal with multiple paths to same state in MCTS?

Many games have multiple paths to the same states. What is the appropriate way to deal with this in MCTS? If the state appears once in the tree, but with multiple parents, then it seems to be ...
Jay McCarthy's user avatar
8 votes
1 answer
2k views

Does AlphaZero use Q-Learning?

I was reading the AlphaZero paper Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm, and it seems they don't mention Q-Learning anywhere. So does AZ use Q-...
Avetik's user avatar
  • 115
8 votes
1 answer
2k views

Any interesting ways to combine Monte Carlo tree search with the minimax algorithm?

I've been working on a game-playing engine for about half a year now, and it uses the well known algorithms. These include minimax with alpha-beta pruning, iterative deepening, transposition tables, ...
Inertial Ignorance's user avatar
8 votes
1 answer
2k views

MCTS: How to choose the final action from the root

When the time allotted to Monte Carlo tree search runs out, what action should be chosen from the root? The original UCT paper (2006) says bestAction in their ...
user76284's user avatar
  • 347
8 votes
2 answers
633 views

How can alpha zero learn if the tree search stops and restarts before finishing a game?

I am trying to understand how alpha zero works, but there is one point that I have problems understanding, even after reading several different explanations. As I understand it (see for example https:/...
Jonathan Lindgren's user avatar
7 votes
2 answers
2k views

Why AlphaGo didn't use Deep Q-Learning?

In the previous research, in 2015, Deep Q-Learning shows its great performance on single player Atari Games. But why do AlphaGo's researchers use CNN + MCTS instead of Deep Q-Learning? is that because ...
malioboro's user avatar
  • 2,819
7 votes
3 answers
350 views

Would AlphaGo Zero become perfect with enough training time?

Would AlphaGo Zero become theoretically perfect with enough training time? If not, what would be the limiting factor? (By perfect, I mean it always wins the game if possible, even against another ...
Christopher King's user avatar
7 votes
1 answer
364 views

How does Hearthstone AI deal with random events

I want to learn a lot about the AI of CCG, such as Hearthstone. And now I have known one of the main algorithms that used in this kind of games, MCTS. It analyses the most promising moves, and expands ...
zen's user avatar
  • 73
7 votes
1 answer
657 views

When does the selection phase exactly end in MCTS?

All sources I can find provide a similar explanation to each phase. In the Selection Phase, we start at the root and choose child nodes until reaching a leaf. Once the leaf is reached (assuming the ...
Ralff's user avatar
  • 173
7 votes
2 answers
506 views

In this implementation of the Information Set Monte Carlo Tree Search, why can't the players see the cards of each other?

After reading this paper about Monte Carlo methods for imperfect information games with elements of uncertainty, I couldn't understand the application of the determinization step in the author's ...
tamirok's user avatar
  • 71
7 votes
0 answers
1k views

How is the rollout from the MCTS implemented in both of the AlphaGo Zero and the AlphaZero algorithms?

In the vanilla Monte Carlo tree search (MCTS) implementation, the rollout is usually implemented following a uniform random policy, that is, it takes random actions until the game is finished and only ...
ihavenoidea's user avatar
7 votes
1 answer
213 views

How could an AI detect whether an enemy in a game can be blocked off/trapped?

Imagine a game played on a 10x10 grid system where a player can move up down left or right and imagine there are two players on this grid: An enemy and you. In this game, there are walls on the grid ...
Ahmed 's user avatar
  • 71
6 votes
2 answers
2k views

Rollout algorithm like Monte Carlo search suggest model based reinforcement learning?

From what I understand, Monte Carlo Tree Search Algorithm is a solution algorithm for model free reinforcement learning (RL). Model free RL means agent doesnt know the transition and reward model. ...
user21872's user avatar
6 votes
1 answer
215 views

How do I know when to use which Monte Carlo method?

I'm a bit confused with extensive number of different Monte Carlo methods such as: Hamiltonian/Hybrid Monte Carlo (HMC), Dynamic Monte Carlo (DMC), Markov chain Monte Carlo (MCMC), Kinetic Monte ...
kenorb's user avatar
  • 10.5k
6 votes
2 answers
2k views

How does AlphaZero's MCTS work when starting from the root node?

From the AlphaGo Zero paper, during MCTS, statistics for each new node are initialized as such: ${N(s_L, a) = 0, W (s_L, a) = 0, Q(s_L, a) = 0, P (s_L, a) = p_a}$. The PUCT algorithm for selecting ...
sb3's user avatar
  • 147
6 votes
1 answer
3k views

When to expand and when to simulate in Monte Carlo Tree Search?

In Monte Carlo Tree Search (MCTS), we start at root node $R$. Then we select some leaf node $L$. And we expand $L$ by one or more child nodes and simulate from the child to the end of the game. When ...
Soroush's user avatar
  • 98
6 votes
1 answer
2k views

What should we do when the selection step selects a terminal state?

In Monte Carlo tree search, what should we do when the selection step selects a terminal state (i.e. a won or lost state), which is, by definition, a leaf node? Expansion and simulation is not in ...
degski's user avatar
  • 163
6 votes
1 answer
2k views

Which nodes are expanded in the expansion phase of MCTS?

I'm confused regarding a specific detail of MCTS. To illustrate my question, let's take the simple example of tic-tac-toe. After the selection phase, when a leaf node is reached, the tree is expanded ...
chessprogrammer's user avatar
6 votes
0 answers
200 views

How exactly does self-play work, and how does it relate to MCTS?

I am working towards using RL to create an AI for a two-player, hidden-information, a turn-based board game. I have just finished David Silver's RL course and Denny Britz's coding exercises, and so am ...
Alienator's user avatar
5 votes
1 answer
1k views

Is the new AlphaGo implementation using Generative Adversarial Networks?

I read through the publication Mastering the game of Go without Human Knowledge. It doesn't seem to use GANs, just a new form of search and reinforcement learning.
dougvk's user avatar
  • 163
5 votes
1 answer
122 views

Why didn't champion of the Go game manage to win the last game against AlphaGo, after winning the 4th one?

In the documentary about the match, it is said that after losing the 4th game, AlphaGo came back stronger and started to play in a weird way (not human-like) and it was pretty impossible to be beaten. ...
Jay Critch's user avatar
5 votes
1 answer
1k views

Which algorithms can we use on games with high branching factors (e.g. Connect6)?

Connect6 is an example of a game with a very high branching factor. It is about 45 thousand, dwarfing even the impressive Go. Which algorithms can we use on games with such high branching factors? I ...
Christopher King's user avatar
5 votes
2 answers
611 views

What is a simple game for validation of MCTS?

What is a simple turn-based game, that can be used to validate a Monte-Carlo Tree Search code and it's parameters? Before applying it to problems where I do not have a possiblity to validate its ...
allo's user avatar
  • 310
5 votes
1 answer
578 views

In MCTS, what to do if I do not want to simulate till the end of the game?

I'm trying to implement MCTS with UCT for a board game and I'm kinda stuck. The state space is quite large (3e15), and I'd like to compute a good move in less than 2 seconds. I already have MCTS ...
Sami's user avatar
  • 53
5 votes
1 answer
1k views

What should the initial UCT value be with MCTS, when leaf's simulation count is zero? Infinity?

I am implenting a Monte Carlo Tree Search algorithm, where the selection process is done through Upper Confidence Bound formula: ...
semyd's user avatar
  • 153
4 votes
1 answer
363 views

Does the AlphaZero algorithm keep the subtree statistics after each move during MCTS?

This question is regarding the Monte Carlo Tree Search (MCTS) algorithm presented in the AlphaZero paper (arXiv). As described in the paper, each MCTS used 800 simulations to determine the next action....
julian's user avatar
  • 43
4 votes
1 answer
379 views

How Does AlphaGo Zero Implement Reinforcement Learning?

AlphaGo Zero (https://deepmind.com/blog/alphago-zero-learning-scratch/) has several key components that contribute to it's success: A Monte Carlo Tree Search Algorithm that allows it to better search ...
SeeDerekEngineer's user avatar
4 votes
1 answer
2k views

How to run a Monte Carlo Tree Search MCTS for stochastic environment?

For MCTS there is an expansion phase where we make a move and list down all the next states. But this is complicated by the fact that for some games, after making the move, there is a stochastic ...
xiaodai's user avatar
  • 141
4 votes
1 answer
650 views

How does Monte Carlo Tree Search UCT exploitation value change based on perspective?

In this blog toward the end, the author writes the following: For the sake of my question, let’s assume that a terminal state gives a reward of +1 for a win and -1 for a loss. When the author says “...
Hanzy's user avatar
  • 519
4 votes
1 answer
743 views

Is there any value given to each chess piece in AlphaZero? [closed]

Recently, DeepMind's AlphaZero chess algorithm did better than the prior best chess software Stockfish. I read the paper Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning ...
Basj's user avatar
  • 151
4 votes
1 answer
368 views

Is the playout started from a leaf or child of leaf in Monte Carlo Tree Search?

On Wikipedia, the MCTS algorithm is described Selection: start from root $R$ and select successive child nodes until a leaf node $L$ is reached. A leaf is any node from which no simulation (playout) ...
Charlie's user avatar
  • 43
4 votes
1 answer
529 views

Why do neural nets and machine learning tend to work well with MCTS, but not with regular Minimax game-playing AI?

I've often heard MCTS grouped together with neural nets and machine learning. From what I gather, MCTS uses a refined intuition (from maching learning) to evaluate positions. This allows it to better ...
Inertial Ignorance's user avatar
3 votes
1 answer
2k views

How fast does Monte Carlo tree search converge?

How fast does Monte Carlo Tree Search converge? Is there a proof that it converges? How does it compare to temporal-difference learning in terms of convergence speed (assuming the evaluation step is a ...
ATidedHumour's user avatar
3 votes
1 answer
719 views

Which Reinforcement Learning algorithms are efficient for episodic problems?

I have some episodic datasets extracted from a turn-based RTS game in which the current actions leading to the next state doesn’t determine the final solution/outcome of the episode. The learning is ...
Zadiq's user avatar
  • 33
3 votes
1 answer
108 views

Do AIs based on MCTS start each game from scratch?

AIs that rely on MCTS - like AlphaGo - create their decision tree as the game progresses. Do they start from scratch each game and build a new tree or do they keep the tree and grow it from game to ...
Demento's user avatar
  • 1,684
3 votes
1 answer
576 views

Can we use MCTS without a generative model?

From what I have understood reading the UCT paper Bandit based monte-carlo planning, by Levente Kocsis and Csaba Szepesvári, MCTS/UCT requires a generative model. Does it mean that, in case there is ...
okkhoy's user avatar
  • 151
3 votes
1 answer
437 views

What is the advantage of using MCTS with value based methods over value based methods only?

I have been trying to understand why MCTS is very important to the performance of RL agents, and the best description I found was from the paper Bootstrapping from Game Tree Search stating: ...
Hossam's user avatar
  • 33
3 votes
1 answer
538 views

How UCT in MCTS selection phase avoids starvation?

The first step of MCTS is to keep choosing nodes based on Upper Confidence Bound applied to trees (UCT) until it reaches a leaf node where UCT is defined as $$\frac{w_i}{n_i}+c\sqrt{\frac{ln(t)}{n_i}},...
user8714896's user avatar
3 votes
1 answer
248 views

AlphaGo Zero: does $Q(s_t, a)$ dominate $U(s_t, a)$ in difficult game states?

AlphaGo Zero AlphaGo Zero uses a Monte-Carlo Tree Search where the selection phase is governed by $\operatorname*{argmax}\limits_a\left( Q(s_t, a) + U(s_t, a) \right)$, where: the exploitation ...
user3667125's user avatar
  • 1,590
3 votes
2 answers
382 views

How can we efficiently and unbiasedly decide which children to generate in the expansion phase of MCTS?

When executing MCTS' expansion phase, where you create a number of child nodes, select one of the numbers, and simulate from that child, how can you efficiently and unbiasedly decide which child(ren) ...
Jay McCarthy's user avatar
3 votes
1 answer
1k views

Several questions related to UCT and MCTS [closed]

In Bandit Based Monte-Carlo Planning, the article where UCT is introduced as a planning algorithm, there is an algorithm description in page 285 (4 of the pdf). Comparing this implementation of UCT (a ...
Miguel Saraiva's user avatar
3 votes
1 answer
292 views

Why is Monte Carlo used as the tree search algorithm for AlphaGo?

Could a better algorithm other than Monte Carlo be used for the AlphaGo computer? Why didn't the DeepMind team think of choosing another kind of algorithm rather than spending time on their neural ...
Jay Critch's user avatar
3 votes
1 answer
51 views

Is it meaningful to give more weight to the result of monte carlo search with less turn win?

I'm programming on Connect6 with MCTS. Monte Carlo Tree Search is based on random moves. It counts up the number of wins in certain moves. (Whether it wins in 3 turns or 30 turns) Is the move with ...
kim's user avatar
  • 31