Skip to main content
21 votes
Accepted

Why does the policy network in AlphaZero work?

The output of the policy network is as described in the original paper: A move in chess may be described in two parts: selecting the piece to move, and then selecting among the legal moves for ...
Neil Slater's user avatar
  • 32.1k
17 votes
Accepted

Why were Chess experts surprised by the AlphaZero's victory against Stockfish?

Good question. First and foremost is that in Go deepmind had no superhuman opponents to challenge. Go engines were not anywhere near the highest level of the top human players. In chess, however, ...
Chris D's user avatar
  • 186
14 votes
Accepted

Is AlphaZero an example of an AGI?

Good question! AlphaZero, though a major milestone, is most definitely not an AGI :) AlphaGo, though strong at the game of Go, is narrowly strong ("strong-narrow AI"), defined as strength in a ...
DukeZhou's user avatar
  • 6,237
10 votes

Does Monte Carlo tree search qualify as machine learning?

John's answer is correct in that MCTS is traditionally not viewed as a Machine Learning approach, but as a tree search algorithm, and that AlphaZero combines this with Machine Learning techniques (...
Dennis Soemers's user avatar
  • 10.3k
9 votes
Accepted

Does Monte Carlo tree search qualify as machine learning?

Monte Carlo Tree Search is not usually thought of as a machine learning technique, but as a search technique. There are parallels (MCTS does try to learn general patterns from data, in a sense, but ...
John Doucette's user avatar
7 votes
Accepted

Does AlphaZero use Q-Learning?

Note: you mentioned in the comments that you are reading the old, pre-print version of the paper describing AlphaZero on arXiv. My answer will be for the "official", peer-reviewed, more recent ...
Dennis Soemers's user avatar
  • 10.3k
6 votes
Accepted

What is the difference between DQN and AlphaGo Zero?

DQN and AlphaZero do not share much in terms of implementation. However, they are based on the same Reinforcement Learning (RL) theoretical framework. If you understand terms like MDP, reward, return, ...
Neil Slater's user avatar
  • 32.1k
6 votes
Accepted

Clarifying representation of Neural Nerwork input for Chess Alpha Zero

For anyone wondering, I believe to have found the answer: Yes, it will be an 8x8 plane where all the entries are the same, the number of moves (or mpves with no progress). There are two repetitions ...
Andrew's user avatar
  • 121
6 votes
Accepted

How does the Alpha Zero's move encoding work?

Let's do the code, so all the details are down. Encoding dictionary: ...
Kostya's user avatar
  • 2,524
5 votes
Accepted

How does AlphaZero use its value and policy heads in conjunction?

I am also a bit confused by your wording but I will try to clear some things up. During MCTS the policy head is used to guide the search while the value head is used as a replacement for roll outs to ...
Ozzi's user avatar
  • 66
5 votes
Accepted

How does AlphaZero's MCTS work when starting from the root node?

I looked at the Python pseudo-code attached to the Data S1 of the Supplementary Materials of the AlphaZero paper. Here is my findings: Contrary to the paper, AlphaZero does not store $\{N(s, a), W(S, ...
user3667125's user avatar
  • 1,570
5 votes
Accepted

Does the AlphaZero algorithm keep the subtree statistics after each move during MCTS?

The supplementary material of the AlphaZero paper states the following: Unless otherwise specified, the training and search algorithm and parameters are identical to AlphaGo Zero. I didn't see any ...
DeepQZero's user avatar
  • 1,379
4 votes

Why were Chess experts surprised by the AlphaZero's victory against Stockfish?

MCTS for chess had been tried in the literature with little success. It was assumed AlphaGo's approach would never work on chess, maybe in Go but not in chess. Suddenly, Google announced the approach ...
SmallChess's user avatar
  • 1,411
4 votes
Accepted

What part of the game is the value network trained to predict a winner on?

To my understanding, this is basically a supervised learning problem, where from the self play we have games associated with their winners, and the network is being trained to map game states to ...
Neil Slater's user avatar
  • 32.1k
4 votes

Would AlphaGo Zero become perfect with enough training time?

Assuming you mean a mathematically perfect player, similar to what we can achieve trivially in Tic Tac Toe, then the answer is "maybe". The underlying reinforcement learning algorithms that it uses do ...
Neil Slater's user avatar
  • 32.1k
4 votes
Accepted

What happens before the first 8 moves in Alpha Zero?

On page 13, right under Table S1 in the linked paper, this is explained (emphasis in bold at the end mine): Each set of planes represents the board position at a time-step $t - T + 1, \dots, t$, ...
Dennis Soemers's user avatar
  • 10.3k
4 votes
Accepted

What knowledge is required for understanding the AlphaZero paper?

The more you read, the more deeply you can understand any paper, but given your stated background, reading the Monte-Carlo Tree Search chapter of Barto & Sutton, plus Gerald Tesauro's TD-Gammon ...
John Doucette's user avatar
4 votes
Accepted

What can we learn from AlphaZero in the development towards AGI?

Some learnings from AlphaZero: Self-play, and more generally sandbox training, is effective. This indicates that given the right enviroment and enough computational power we can build highly ...
Rexcirus's user avatar
  • 1,174
4 votes
Accepted

Deep Q Networks v Monte Carlo Tree Search in Alpha Zero

DQN and MCTS (+ NNs) are algorithms of two distinct RL flavors: DQN is model-free RL: it's able to learn the value of actions without having access to the underlying MDP (i.e. environment's model), ...
Luca Anzalone's user avatar
4 votes

Would AlphaZero perform better if made with transformers?

Yes! LeelaChessZero, an open-source re-implementation and continuation of AlphaZero, has been experimenting with this for a while now. Their strongest networks are currently transformers, not ...
KarelPeeters's user avatar
3 votes
Accepted

Can AlphaZero considered as Multi-Agent Deep Reinforcement Learning?

Depends on perspective. On one hand, you have an agent playing in an environment with another agent also evolving. This falls under the definition of Multi-Agent Learning, as can be seen with works ...
BlueMoon93's user avatar
3 votes
Accepted

When does AlphaZero play suboptimal moves?

During the self-play training process, AlphaZero does not greedily play only the moves it thinks are "best" (which would normally be the move with the highest visit count leading out of the root node ...
Dennis Soemers's user avatar
  • 10.3k
3 votes
Accepted

What does it mean for AlphaZero's network to be "fully trained"

Neural network will eventually reach limit of it's approximation power. You can't expect to learn more and more things infinitely long with the same amount of learnable parameters. Also, if you ...
Brale's user avatar
  • 2,386
3 votes

Alpha Zero queen promotion

It means that there is no explicit coding of action choices to promote to queen, it is the default assumption if the underpromotion actions are not taken. The Alpha Zero chess implementation can ...
Neil Slater's user avatar
  • 32.1k
3 votes
Accepted

How can I use one neural network for both players in Alpha Zero (Connect 4)?

Let's define your problem from another point of view. Let's say that in this RL problem you have two agents (agent1 and agent2) that compete with each other in order to accomplish their own goal, i.e.,...
rjgp's user avatar
  • 46
3 votes
Accepted

Is there any value given to each chess piece in AlphaZero?

is there a value given for each piece (e.g. 1 for pawn, 3 for knight, 9 for queen, etc.) to train the algorithm, or does the algorithm learn this by himself? No, there are no such explicit values ...
Dennis Soemers's user avatar
  • 10.3k
3 votes

Combining deep reinforcement learning with alpha-beta pruning

Yes it's possible to to combine AlphaZero with Minimax methods (including alpha-beta pruning). AlphaZero itself is combination of Monte Carlo Tree Search (MCTS) and Deep Network, where MCTS is used ...
mirror2image's user avatar
3 votes
Accepted

Would AlphaGo Zero become perfect with enough training time?

We cannot tell with certainty whether AlphaGo Zero would become perfect with enough training time. This is because none of the parts (Neural Network) that would benefit from infinite training time (= ...
Dennis Soemers's user avatar
  • 10.3k
3 votes
Accepted

Is it practical to train AlphaZero or MuZero (for indie games) on a personal computer?

The vast majority of neural networks are now trained on graphics processing units (GPUs) or specialised accelerator hardware such as tensor processing units (TPUs). In Mastering Chess and Shogi by ...
htl's user avatar
  • 1,010
3 votes

Why does Alpha Zero's Neural Network flip the board to be oriented towards the current player?

I am not an expert in RL. I have been playing Go for some years. Let's quote from AlphaZero's paper first: Aside from komi, the rules of Go are also invariant to colour transposition; this knowledge ...
loct's user avatar
  • 131

Only top scored, non community-wiki answers of a minimum length are eligible