Questions tagged [monte-carlo-tree-search]

For questions related to Monte Carlo Tree Search (MCTS), which is a best-first, rollout-based tree search algorithm. MCTS gradually improves its evaluations of nodes in the trees using (semi-)random rollouts through those nodes, focusing a larger proportion of rollouts on the parts of the tree that are the most promising.

Filter by
Sorted by
Tagged with
2
votes
1answer
30 views

How can I develop an MCTS-like algorithm for program induction?

I'd like to develop an MCTS-like (Monte Carlo Tree Search) algorithm for program induction, i.e. learning programs from examples. My initial plan is for nodes to represent programs and for the search ...
0
votes
0answers
11 views

Train a Neuronal Network with MCTS Data

I failed using PPO to train a multiplayer card game. Thus I tested monte carlo tree search (mcts) to predict good moves. This works now (you can test the game here. As calculating a good move using ...
3
votes
0answers
30 views

How exactly does self-play work, and how does it relate to MCTS?

I am working towards using RL to create an AI for a two-player, hidden-information, a turn-based board game. I have just finished David Silver's RL course and Denny Britz's coding exercises, and so am ...
6
votes
1answer
77 views

MCTS: How to choose the final action from the root

When the time allotted to Monte Carlo tree search runs out, what action should be chosen from the root? The original UCT paper (2006) says bestAction in their ...
4
votes
2answers
213 views

Determinization step in Information Set Monte Carlo Tree Search

After reading this paper about Monte Carlo methods for imperfect information games with elements of uncertainty, I couldn't understand the application of determinization step in author's ...
9
votes
3answers
532 views

Does Monte Carlo tree search qualify as machine learning?

To the best of my understanding, the Monte Carlo tree search (MCTS) algorithm is an alternative to minimax for searching a tree of nodes. It works by choosing a move (generally, the one with the ...
4
votes
1answer
39 views

How could an AI detect whether an enemy in a game can be blocked off/trapped?

Imagine a game played on a 10x10 grid system where a player can move up down left or right and imagine there are two players on this grid: An enemy and you. In this game, there are walls on the grid ...
15
votes
3answers
5k views

How do I choose the best algorithm for a board game like checkers?

How do I choose the best algorithm for a board game like checkers? So far, I have considered only three algorithms, namely, minimax, alpha-beta pruning, and Monte Carlo tree search (MCTS). Apparently,...
0
votes
0answers
19 views

Formulating MCTS with random outcomes of actions?

I am working on implementing MCTS for a scheduling problem where MCTS is formulated each time there are multiple jobs that need to be scheduled. When a job is executed, the resulting state of the ...
1
vote
2answers
78 views

How to understand the 4 steps of Monte Carlo Tree Search

From many blogs and this one https://web.archive.org/web/20160308070346/http://mcts.ai/about/index.html We know that the process of MCTS algorithm has 4 steps. Selection: Starting at root node R,...
3
votes
0answers
82 views

Understanding an execution of the Monte Carlo tree search algorithm

I have the execution of the Monte Carlo Tree Search (MCTS) below. I need to expand it, but I don't understand steps 1 and 2. Why does it go to the first node and then make a new node, instead of ...
2
votes
0answers
86 views

What is the time complexity of an unparellelized Monte Carlo tree search?

I am writing a report where I used a slightly modified version of MCTS (not parallelized). I thought It could be interesting if I could calculate its time complexity. I'd appreciate any help I could ...
1
vote
0answers
43 views

Proof of Correctness of Monte Carlo Tree Search

I'm trying to write the proof of correctness of Monte Carlo Tree Search. Any help would be really appreciated.
6
votes
2answers
224 views

MCTS for non-deterministic games with very high branching factor for chance nodes

I'm trying to use a Monte Carlo Tree Search for a non-deterministic game. Apparently, one of the standard approaches is to model non-determinism using chance nodes. The problem for this game is that ...
3
votes
1answer
147 views

Is the playout started from a leaf or child of leaf in Monte Carlo Tree Search?

On Wikipedia, the MCTS algorithm is described Selection: start from root $R$ and select successive child nodes until a leaf node $L$ is reached. A leaf is any node from which no simulation (playout)...
4
votes
1answer
866 views

When to expand and when to simulate in Monte Carlo Tree Search?

In Monte Carlo Tree Search (MCTS), we start at root node $R$. Then we select some leaf node $L$. And we expand $L$ by one or more child nodes and simulate from the child to the end of the game. When ...
6
votes
1answer
121 views

When does the selection phase exactly end in MCTS?

All sources I can find provide a similar explanation to each phase. In the Selection Phase, we start at the root and choose child nodes until reaching a leaf. Once the leaf is reached (assuming the ...
3
votes
2answers
102 views

How can we efficiently and unbiasedly decide which children to generate in the expansion phase of MCTS?

When executing MCTS' expansion phase, where you create a number of child nodes, select one of the numbers, and simulate from that child, how can you efficiently and unbiasedly decide which child(ren) ...
5
votes
1answer
636 views

Which nodes are expanded in the expansion phase of MCTS?

I'm confused regarding a specific detail of MCTS. To illustrate my question, let's take the simple example of tic-tac-toe. After the selection phase, when a leaf node is reached, the tree is expanded ...
2
votes
1answer
397 views

Can we use MCTS without a generative model?

From what I have understood reading the UCT paper Bandit based monte-carlo planning, by Levente Kocsis and Csaba Szepesvári, MCTS/UCT requires a generative model. Does it mean that, in case there is ...
5
votes
1answer
766 views

What should we do when the selection step selects a terminal state?

In Monte Carlo tree search, what should we do when the selection step selects a terminal state (i.e. a won or lost state), which is, by definition, a leaf node? Expansion and simulation is not in ...
1
vote
2answers
335 views

Is Monte Carlo Tree Search appropriate for problems with large state and action spaces?

I'm doing a research on a finite-horizon Markov decision process with $t=1, \dots, 40$ periods. In every time step $t$, the (only) agent has to chose an action $a(t) \in A(t)$, while the agent is in ...
3
votes
0answers
80 views

How is the rollout from the MCTS implemented in both of the AlphaGo Zero and the AlphaZero algorithms?

In a vanilla Monte Carlo tree search (MCTS) implementation, the rollout is usually implemented following a uniform random policy, that is, it takes random actions until the game is finished and only ...
1
vote
0answers
58 views

How is q-learning related to game trees?

At a first look, q-learning is a revolutionary strategy in realizing Artificial Intelligence. It has to do with finding a policy, a reward structure, neural networks for storing the q-table and a ...
0
votes
0answers
20 views

Tweaking MCTS to account for opponent knowing the state of the game

I'm making an artificial intelligence for a card game using MCTS. With a standard 52-cards deck, 4 hands are dealt: 1 for each of the 3 players and one extra hand. Then, each player gets the choice to ...
3
votes
2answers
89 views

What is the appropriate way to deal with multiple paths to same state in MCTS?

Many games have multiple paths to the same states. What is the appropriate way to deal with this in MCTS? If the state appears once in the tree, but with multiple parents, then it seems to be ...
3
votes
1answer
121 views

Several questions related to UCT and MCTS

In Bandit Based Monte-Carlo Planning, the article where UCT is introduced as a planning algorithm, there is an algorithm description in page 285 (4 of the pdf). Comparing this implementation of UCT (...
6
votes
1answer
194 views

Does AlphaZero use Q-Learning?

I was reading the AlphaZero paper Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm, and it seems they don't mention Q-Learning anywhere. So does AZ use Q-...
5
votes
2answers
125 views

What is a simple game for validation of MCTS?

What is a simple turn-based game, that can be used to validate a Monte-Carlo Tree Search code and it's parameters? Before applying it to problems where I do not have a possiblity to validate its ...
3
votes
1answer
84 views

Why is Monte Carlo used as the tree search algorithm for AlphaGo?

Could a better algorithm other than Monte Carlo be used for the AlphaGo computer? Why didn't the DeepMind team think of choosing another kind of algorithm rather than spending time on their neural ...
1
vote
1answer
199 views

How fast does Monte Carlo tree search converge?

How fast does Monte Carlo Tree Search converge? Is there proof that it does converge? How does it compare to Temporal Difference learning in terms of convergence speed (assuming the evaluation step ...
7
votes
2answers
92 views

How can alpha zero learn if the tree search stops and restarts before finishing a game?

I am trying to understand how alpha zero works, but there is one point that I have problems understanding, even after reading several different explanations. As I understand it (see for example https:/...
3
votes
1answer
113 views

How Does AlphaGo Zero Implement Reinforcement Learning?

AlphaGo Zero (https://deepmind.com/blog/alphago-zero-learning-scratch/) has several key components that contribute to it's success: A Monte Carlo Tree Search Algorithm that allows it to better search ...
4
votes
1answer
234 views

Any interesting ways to combine Monte Carlo tree search with the minimax algorithm?

I've been working on a game-playing engine for about half a year now, and it uses the well known algorithms. These include minimax with alpha-beta pruning, iterative deepening, transposition tables, ...
2
votes
0answers
13 views

How is GARB implemented in PGRD-DL to calculate gradients w.r.t. internal rewards?

In section 3 of this paper the author outlines how GARB was adapted to reduce the variance in updating parameters to an internal reward function estimator. I have read it a number of times and ...
3
votes
1answer
25 views

How does Monte Carlo Tree Search UCT exploitation value change based on perspective?

In this blog toward the end, the author writes the following: For the sake of my question, let’s assume that a terminal state gives a reward of +1 for a win and -1 for a loss. When the author says “...
3
votes
0answers
55 views

Feature Selection using Monte Carlo Tree Search

I'm trying to tackle the problem of feature selection as an RL problem, inspired by the paper Feature Selection as a One-Player Game. I know Monte-Carlo tree search (MCTS) is hardly RL. So, I used ...
2
votes
1answer
58 views

Similarities and differences between UCT algorithms in (i), (ii), (iii) and (iv)?

I am trying to understand the similarities and differences between: (i) the UCT algorithm in Kocsis and Szepesvári (2006); (ii) the UCT algorithm in Section 3.3 of Browne et al (2012); (iii) the MCTS ...
2
votes
1answer
81 views

Chess Engine for low clock cycle CPU

I'm looking for a simple chess algorithm that works well on a ~132MHz CPU. I know that MiniMax would take too long to go deep in order to find good moves. Is Monte Carlo Tree Search working well on ...
4
votes
2answers
358 views

Rollout algorithm like Monte Carlo search suggest model based reinforcement learning?

From what I understand, Monte Carlo Tree Search Algorithm is a solution algorithm for model free reinforcement learning (RL). Model free RL means agent doesnt know the transition and reward model. ...
3
votes
1answer
155 views

Why do neural nets and machine learning tend to work well with MCTS, but not with regular Minimax game-playing AI?

I've often heard MCTS grouped together with neural nets and machine learning. From what I gather, MCTS uses a refined intuition (from maching learning) to evaluate positions. This allows it to better ...
5
votes
1answer
85 views

How does Hearthstone AI deal with random events

I want to learn a lot about the AI of CCG, such as Hearthstone. And now I have known one of the main algorithms that used in this kind of games, MCTS. It analyses the most promising moves, and expands ...
1
vote
1answer
90 views

How do you generate the transition probabilities of a non-trivial MDP?

I understand an MDP (Markov Decision Process) model is a tuple of $\{S, A, P, R \}$ where: $S$ is a discrete set of states $A$ is a discrete set of actions $P$ is the transition matrix ie. $P(s' \mid ...
3
votes
1answer
92 views

Weighted move rating for AI

My AI (for the card game schnapsen) currently calculates every possible way the game could end and then evaluates the percentage of winning for every playable card / move. The calculation is done ...
2
votes
1answer
37 views

Is it meaningful to give more weight to the result of monte carlo search with less turn win?

I'm programming on Connect6 with MCTS. Monte Carlo Tree Search is based on random moves. It counts up the number of wins in certain moves. (Whether it wins in 3 turns or 30 turns) Is the move with ...
7
votes
3answers
939 views

Why does Monte Carlo work when a real opponent's behavior may not be random

I am learning about Monte Carlo algorithms and struggling to understand the following: If simulations are based on random moves, how can the modeling of the opponent's behavior work well? For ...
3
votes
1answer
59 views

Should I use Monte Carlo or a classifier for this Decision Making problem?

I want to build a model to support decision making for loan insurance proposal. There are three actors in the problem: a bank, a loaner applicant (someone who ask for a loan) and a counselor. The ...
6
votes
3answers
190 views

Would AlphaGo Zero become perfect with enough training time?

Would AlphaGo Zero become theoretically perfect with enough training time? If not, what would be the limiting factor? (By perfect, I mean it always wins the game if possible, even against another ...
4
votes
1answer
413 views

Algorithms for games with very high branching factors (Connect6)

Connect6 is an example of a game with a very high branching factor. It is about 45 thousand, dwarfing even the impressive Go. What algorithms can you use on games with such high branching factors? I ...
3
votes
1answer
425 views

AlphaZero chess algorithm, Monte Carlo search

Recently, DeepMind's AlphaZero chess algorithm did better than the prior best chess software Stockfish. I read an arxiv paper about it but I'm not sure if: is there a value given for each piece (e.g. ...