Skip to main content
Share Your Experience: Take the 2024 Developer Survey

Questions tagged [alphazero]

For questions related to DeepMind's AlphaZero, which is a computer program that can play Go, Chess, and Shogi. AlphaZero achieved, within 24 hours of training, a superhuman level of play in these three games by defeating world-champion programs Stockfish, Elmo, and the 3-day version of AlphaGo Zero. AlphaZero was introduced in "Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm" (2017) by David Silver et al.

Filter by
Sorted by
Tagged with
0 votes
1 answer
32 views

Could AlphaZero be trained to prefer "beautiful" games?

Could a version of AlphaZero be trained that learned not only how to win, but how to win in a "beautiful" way? Jurgen Schmidhuber wrote a paper in 2008, which basically models "beauty&...
swami's user avatar
  • 101
1 vote
1 answer
181 views

Would AlphaZero perform better if made with transformers?

AlphaZero utilized a residual convolutional neural network to estimate move policy and position value. If it was rebuilt today, would it be more efficient and powerful if they used a transformer ...
Ben G's user avatar
  • 123
3 votes
1 answer
96 views

AlphaZero Chess: High portion of "draws" during first rounds of self-play hampers learning

I am experimenting with a self-programmed version of AZ Chess following the described methodology in the official paper. I am experiencing that at the beginning of the self-play (when the ANN weights ...
Oliver Ewald's user avatar
2 votes
1 answer
105 views

Why does training a NN using MCTS work even if the number of simulations isn't much larger than the number of actions?

tl;dr If the visit rates of children generated by MCTS is biased because not enough samples were taken, why doesn't the network learn random behavior? My understanding of combining MCTS and NNs (e.g. ...
Christopher's user avatar
0 votes
0 answers
61 views

How are Target Values Generated in Alpha Zero Architecture

I am a little confused as to how the target values are generated to train the neural network with the Alpha Zero architecture(in specific to a chess game). I understand how the improved policy is ...
Kiran Manicka's user avatar
1 vote
1 answer
466 views

Deep Q Networks v Monte Carlo Tree Search in Alpha Zero

Recently I've been studying how Deep Q Networks work, and as I was reading I just assumed that game engines like Alpha Zero use Deep Q Learning to choose actions. But as I was reading the Alpha Zero ...
Kiran Manicka's user avatar
0 votes
0 answers
91 views

Policy Value network predictions in Alpha Zero with ranked rewards

So I have been trying to implement the ranked rewards (R2) algorithm from the paper "Ranked Reward: Enabling Self-Play Reinforcement Learning for Combinatorial Optimization" http://arxiv.org/...
Darkdragon84's user avatar
1 vote
1 answer
351 views

Reproducing AlphaZero/MuZero: Failed to beat initial model in arena

I am trying to reproduce AlphaZero's algorithm on the board game Carcassonne. Since I want to use the final game score differences (i.e. victory point of player 1 - victory point of player 2) as the ...
TommyX's user avatar
  • 13
0 votes
1 answer
76 views

Is AlphaZero's output (action probabilities) vector suboptimal?

The AlphaZero research team states A move in chess may be described in two parts: selecting the piece to move, and then selecting among the legal moves for that piece. We represent the policy π(a|s) ...
ace.exe's user avatar
0 votes
0 answers
88 views

Why do policy gradient methods not work for imperfect information games?

I've heard before that policy gradient and Q-learning approaches fail on games of imperfect information. I was watching this video (starting at 23:45) about the Player of Games AI, but I couldn't ...
JacKeown's user avatar
  • 125
2 votes
1 answer
200 views

Why does AlphaZero not use vanilla MCTS?

I understand main difference between AlphaZero and the classic Monte Carlo tree search is the playout (simulation) step is replaced with a neural network prediction which itself is trained from the ...
user2309803's user avatar
1 vote
1 answer
79 views

When do we use the neural network to predict value during the expansion stage of MCTS in the AlphaZero algorithm?

According to what I understand from the AlphaZero algorithm, a neural network is used to set value and prior probability for a node during the expansion stage of MCTS. On the other hand, according to ...
Mahdi Hosseini's user avatar
1 vote
1 answer
273 views

Neural Network output for the game of Checkers

I'm trying to train a RL agent to play the game of checkers (AlphaZero style) and so far I've managed a proof of concept training a connect 4 agent up until perfection. However, unlike connect 4, ...
Anik Patel's user avatar
2 votes
1 answer
2k views

The reason behind using MCTS over Alpha Beta Pruning in Alphazero

I am not really satisfied with the available analysis of why AlphaZero uses MCTS instead of Alpha Beta search. Some analysis claim that its because MCTS is a lot more humanlike. I disagree because I ...
Dimanjan's user avatar
  • 123
4 votes
1 answer
353 views

Does the AlphaZero algorithm keep the subtree statistics after each move during MCTS?

This question is regarding the Monte Carlo Tree Search (MCTS) algorithm presented in the AlphaZero paper (arXiv). As described in the paper, each MCTS used 800 simulations to determine the next action....
julian's user avatar
  • 43
1 vote
1 answer
151 views

What can we learn from AlphaZero in the development towards AGI?

According to DeepMind, AlphaZero's creative insights coupled with the encouraging results we see in other projects such as AlphaFold, give us confidence in our mission to create general purpose ...
zjeffer's user avatar
  • 143
3 votes
1 answer
285 views

How can AlphaZero be used in other industries besides gaming?

I'm an AI Engineering student from Belgium and I'm writing my bachelor thesis on the creation of a chess computer with deep reinforcement learning based on AlphaZero. My implementation can be found ...
zjeffer's user avatar
  • 143
2 votes
1 answer
178 views

Can AlphaZero develop significantly different playing styles (depending on the random games from which it learrns)?

There is a quite popular video analysing a chess game AlphaZero vs. AlphaZero, called "the perfect game". It leaves some questions open and I'd like to ask them here: Did the two copies of ...
Hans-Peter Stricker's user avatar
1 vote
0 answers
84 views

Examples of rationalizable AI

The marvelous book Game Changer: AlphaZero's Groundbreaking Chess Strategies and the Promise of AI gave rise to this question. It is - in my opinion - a perfect example of rationalizing a piece of AI ...
Hans-Peter Stricker's user avatar
2 votes
3 answers
447 views

Does AlphaGo play random moves in a real competition?

Alphago and AlphaGo zero use random play to generate data and use the data to train DNN. "Random play" means that there is a positive probability for AlphaGo to play some suboptimal moves ...
High GPA's user avatar
  • 163
1 vote
1 answer
411 views

What is a policy training target in AlphaZero?

In AlphaZero's attached pseudocode, they create a training target for the policy network in this way. ...
Druudik's user avatar
  • 149
1 vote
2 answers
606 views

Would AlphaZero work just with a value network?

There is a nice post about the intuition why AlphaZero works. One of the advantages of using a policy network in the games where a perfect simulator is available (such as chess) is to save computation ...
Druudik's user avatar
  • 149
3 votes
1 answer
780 views

What does it mean there is no rollout in AlphaZero's training?

According to a blog post by DeepMind, AlphaZero doesn't have a real rollout. AlphaGo Zero does not use "rollouts" - fast, random games used by other Go programs to predict which player will ...
Daniel's user avatar
  • 165
3 votes
1 answer
395 views

At what point are MCTS results discarded in AlphaZero Training?

Regarding the AlphaZero paper, it is not clear to me when the Monte Carlo Tree Search (MCTS) results will be cleaned up. I assume this has to happen at some point, since mixing results could lead to ...
Daniel's user avatar
  • 165
4 votes
1 answer
771 views

How does policy network learn in AlphaZero?

I'm currently trying to understand how AlphaZero works. There is one thing with the training of the AlphaZero's policy head that confuses me. Basically, in AlphaGo Zero's paper (where the major part ...
Alberto M's user avatar
3 votes
3 answers
1k views

Why does Alpha Zero's Neural Network flip the board to be oriented towards the current player?

While reading the AlphaZero paper in preparation to code my own RL algorithm to play Chess decently well, I saw that the "The board is oriented to the perspective of the current player." I ...
Akshay Ghosh's user avatar
6 votes
1 answer
2k views

How does the Alpha Zero's move encoding work?

I am a beginner in AI. I'm trying to train a multi-agent RL algorithm to play chess. One issue that I ran into was representing the action space (legal moves/or honestly just moves in general) ...
Akshay Ghosh's user avatar
5 votes
1 answer
800 views

Clarifying representation of Neural Nerwork input for Chess Alpha Zero

In the Alpha Zero paper (https://arxiv.org/pdf/1712.01815.pdf) page 13, the input for the NN is described. In the beggining of the page, the authors state that: "The input to the Neural Network ...
Andrew's user avatar
  • 121
5 votes
2 answers
2k views

Is it practical to train AlphaZero or MuZero (for indie games) on a personal computer?

Is it practical/affordable to train an AlphaZero/MuZero engine using a residential gaming PC, or would it take thousands of years of training for the AI to learn enough to challenge humans? I'm having ...
Luke W's user avatar
  • 53
2 votes
0 answers
485 views

What is the consensus on the "correct" temperature settings for the AlphaZero algorithm?

In the AlphaZero learning algorithm, during self-play to generate training games, the move played is chosen with probability proportional to the MCTS visits raised to the $\tau$-th power, where $\tau$ ...
davik's user avatar
  • 121
5 votes
1 answer
677 views

Do AlphaZero/MuZero learn faster in terms of number of games played than humans?

I don't know much about AI and am just curious. From what I read, AlphaZero/MuZero outperform any human chess player after a few hours of training. I have no idea how many chess games a very talented ...
220284's user avatar
  • 153
0 votes
1 answer
720 views

Alpha Zero does not converge for Connect 6, a game with huge branching factor - why?

I have a problem with applying alpha zero self-play to a game (Connect 6) with a huge branching factor (30,000 on average). I have implemented the MCTS as described but I found that during the MCTS ...
javaPhobic's user avatar
6 votes
2 answers
2k views

How does AlphaZero's MCTS work when starting from the root node?

From the AlphaGo Zero paper, during MCTS, statistics for each new node are initialized as such: ${N(s_L, a) = 0, W (s_L, a) = 0, Q(s_L, a) = 0, P (s_L, a) = p_a}$. The PUCT algorithm for selecting ...
sb3's user avatar
  • 147
1 vote
0 answers
61 views

Are inputs into AlphaZero the same during the evaluate step in MCTS and during test time?

From the AlphaZero paper: The input to the neural network is an N × N × (M T + L) image stack that represents state using a concatenation of T sets of M planes of size N × N . Each set of planes ...
sb3's user avatar
  • 147
1 vote
1 answer
163 views

Is there a training data capacity limit for AlphaZero (Chess)?

In AlphaZero, we collect ($s_t, \pi_t, z_t$) tuples from self-play, where $s_t$ is the board state, $\pi_t$ is the policy, and $z_t$ is the reward from winning/losing the game. In other DeepRL off-...
sb3's user avatar
  • 147
3 votes
0 answers
501 views

Stack of Planes as the Action Space Representation for AlphaZero (Chess)

I have a question regarding the action space of the policy network used in AlphaZero. From the paper: We represent the policy π(a|s) by a 8 × 8 × 73 stack of planes encoding a probability ...
sb3's user avatar
  • 147
2 votes
1 answer
161 views

In AlphaZero, do we need to store the data of terminal states?

I have a question about the training data used during the update/back-propagation step of the neural network in AlphaZero. From the paper: The data for each time-step $t$ is stored as ($s_t, \pi_t, ...
sb3's user avatar
  • 147
1 vote
0 answers
472 views

What would be the AlphaGo's performance in continuous action space?

During my research for Google DeepMind's Go-playing program Alpha Go and its successor Alpha Go Zero, I discovered that the system uses a clever pipeline and an interplay of blocks of both policy and ...
maven's user avatar
  • 31
3 votes
1 answer
242 views

AlphaGo Zero: does $Q(s_t, a)$ dominate $U(s_t, a)$ in difficult game states?

AlphaGo Zero AlphaGo Zero uses a Monte-Carlo Tree Search where the selection phase is governed by $\operatorname*{argmax}\limits_a\left( Q(s_t, a) + U(s_t, a) \right)$, where: the exploitation ...
user3667125's user avatar
  • 1,570
1 vote
1 answer
82 views

Is which sense was AlphaGo "just given a rule book"?

I was told that AlphaGo (or some related program) was not explicitly taught even the rules of Go -- if it was "just given the rulebook", what does this mean? Literally, a book written in ...
releseabe's user avatar
  • 141
2 votes
2 answers
107 views

What happens when an opponent a neural network is playing with does not obey the rules of the game (i.e. cheats)?

For example, if AlphaZero plays with an opponent who has a right to move chess figures any way she wants, or make more than 1 move in a turn? Will a neural network adapt to that, as it adapted to an ...
ivan866's user avatar
  • 129
1 vote
2 answers
564 views

In Alpha(Go)Zero, why is the policy extracted from MCTS better than the network one?

I've read through the Alpha(Go)Zero paper and there is only one thing I don't understand. The paper on page 1 states: The MCTS search outputs probabilities π of playing each move. These search ...
Euclid's user avatar
  • 43
3 votes
1 answer
329 views

Can AlphaZero considered as Multi-Agent Deep Reinforcement Learning?

Can AlphaZero considered as Multi-Agent Deep Reinforcement Learning? I could not find a clear answer on this. I would say yes it is Multi Agent Learning, as there are two Agents playing against each ...
flexw's user avatar
  • 33
2 votes
0 answers
268 views

What kind of policy evaluation and policy improvement AlphaGo, AlphaGo Zero and AlphaZero are using

I'm trying to find out what kind of policy improvement and policy evaluation AlphaGo, AlphaGo Zero, and AlphaZero are using. By looking into their respective paper and SI, I can conclude that it is a ...
Daniel Wiczew's user avatar
3 votes
0 answers
80 views

Where does reinforcement learning actually show up in Deepmind's game engines?

From the brief research I've done on the topic, it appears that the way Deepmind's Alphazero or Muzero makes decisions is through Monte Carlo tree searches, where in the randomized simulations allows ...
Amar Srivastava's user avatar
3 votes
0 answers
205 views

Is Monte Carlo tree search needed in partially observable environments during gameplay?

I understand that with a fully observable environment (chess / go etc) you can run an MCTS with an optimal policy network for future planning purposes. This will allow you to pick actions for gameplay,...
Yohahn Ribeiro's user avatar
1 vote
0 answers
139 views

How to encode board before input into the neural net?

Currently I'm working on an educational project (implementation of AlphaZero approach to different types of board games). My biggest concern at the moment is how to encode board before input into the ...
Taissa's user avatar
  • 63
4 votes
1 answer
168 views

In AlphaZero, which policy is saved in the dataset, and how is the move chosen?

I've been doing some research on the principles behind AlphaZero. Especially this cheat sheet (1) and this implementation (2) (in Connect 4) were very useful. Yet, I still have two important questions:...
Jonas De Schouwer's user avatar
3 votes
1 answer
505 views

Building 'evaluation' neural networks for go, reversi, checkers etc, how to train?

I'm trying to build neural networks for games like Go, Reversi, Othello, Checkers, or even tic-tac-toe, not by calculating a move, but by making them evaluate a position. The input is any board ...
RocketNuts's user avatar
7 votes
0 answers
1k views

How is the rollout from the MCTS implemented in both of the AlphaGo Zero and the AlphaZero algorithms?

In the vanilla Monte Carlo tree search (MCTS) implementation, the rollout is usually implemented following a uniform random policy, that is, it takes random actions until the game is finished and only ...
ihavenoidea's user avatar