19 votes
Accepted

Why does the policy network in AlphaZero work?

The output of the policy network is as described in the original paper: A move in chess may be described in two parts: selecting the piece to move, and then selecting among the legal moves for ...
  • 24.6k
16 votes
Accepted

Why were Chess experts surprised by the AlphaZero's victory against Stockfish?

Good question. First and foremost is that in Go deepmind had no superhuman opponents to challenge. Go engines were not anywhere near the highest level of the top human players. In chess, however, ...
  • 176
14 votes
Accepted

Is AlphaZero an example of an AGI?

Good question! AlphaZero, though a major milestone, is most definitely not an AGI :) AlphaGo, though strong at the game of Go, is narrowly strong ("strong-narrow AI"), defined as strength in a ...
  • 6,107
10 votes

Does Monte Carlo tree search qualify as machine learning?

John's answer is correct in that MCTS is traditionally not viewed as a Machine Learning approach, but as a tree search algorithm, and that AlphaZero combines this with Machine Learning techniques (...
  • 9,509
9 votes
Accepted

Does Monte Carlo tree search qualify as machine learning?

Monte Carlo Tree Search is not usually thought of as a machine learning technique, but as a search technique. There are parallels (MCTS does try to learn general patterns from data, in a sense, but ...
7 votes
Accepted

Does AlphaZero use Q-Learning?

Note: you mentioned in the comments that you are reading the old, pre-print version of the paper describing AlphaZero on arXiv. My answer will be for the "official", peer-reviewed, more recent ...
  • 9,509
6 votes
Accepted

What is the difference between DQN and AlphaGo Zero?

DQN and AlphaZero do not share much in terms of implementation. However, they are based on the same Reinforcement Learning (RL) theoretical framework. If you understand terms like MDP, reward, return, ...
  • 24.6k
6 votes
Accepted

Clarifying representation of Neural Nerwork input for Chess Alpha Zero

For anyone wondering, I believe to have found the answer: Yes, it will be an 8x8 plane where all the entries are the same, the number of moves (or mpves with no progress). There are two repetitions ...
5 votes
Accepted

How does the Alpha Zero's move encoding work?

Let's do the code, so all the details are down. Encoding dictionary: ...
  • 1,833
4 votes

Why were Chess experts surprised by the AlphaZero's victory against Stockfish?

MCTS for chess had been tried in the literature with little success. It was assumed AlphaGo's approach would never work on chess, maybe in Go but not in chess. Suddenly, Google announced the approach ...
  • 1,390
4 votes

Would AlphaGo Zero become perfect with enough training time?

Assuming you mean a mathematically perfect player, similar to what we can achieve trivially in Tic Tac Toe, then the answer is "maybe". The underlying reinforcement learning algorithms that it uses do ...
  • 24.6k
4 votes
Accepted

What happens before the first 8 moves in Alpha Zero?

On page 13, right under Table S1 in the linked paper, this is explained (emphasis in bold at the end mine): Each set of planes represents the board position at a time-step $t - T + 1, \dots, t$, ...
  • 9,509
4 votes
Accepted

What knowledge is required for understanding the AlphaZero paper?

The more you read, the more deeply you can understand any paper, but given your stated background, reading the Monte-Carlo Tree Search chapter of Barto & Sutton, plus Gerald Tesauro's TD-Gammon ...
4 votes
Accepted

How does AlphaZero use its value and policy heads in conjunction?

I am also a bit confused by your wording but I will try to clear some things up. During MCTS the policy head is used to guide the search while the value head is used as a replacement for roll outs to ...
  • 56
4 votes
Accepted

Does the AlphaZero algorithm keep the subtree statistics after each move during MCTS?

The supplementary material of the AlphaZero paper states the following: Unless otherwise specified, the training and search algorithm and parameters are identical to AlphaGo Zero. I didn't see any ...
  • 1,027
3 votes
Accepted

When does AlphaZero play suboptimal moves?

During the self-play training process, AlphaZero does not greedily play only the moves it thinks are "best" (which would normally be the move with the highest visit count leading out of the root node ...
  • 9,509
3 votes
Accepted

What does it mean for AlphaZero's network to be "fully trained"

Neural network will eventually reach limit of it's approximation power. You can't expect to learn more and more things infinitely long with the same amount of learnable parameters. Also, if you ...
  • 2,236
3 votes

Alpha Zero queen promotion

It means that there is no explicit coding of action choices to promote to queen, it is the default assumption if the underpromotion actions are not taken. The Alpha Zero chess implementation can ...
  • 24.6k
3 votes
Accepted

How can I use one neural network for both players in Alpha Zero (Connect 4)?

Let's define your problem from another point of view. Let's say that in this RL problem you have two agents (agent1 and agent2) that compete with each other in order to accomplish their own goal, i.e.,...
  • 46
3 votes
Accepted

Can AlphaZero considered as Multi-Agent Deep Reinforcement Learning?

Depends on perspective. On one hand, you have an agent playing in an environment with another agent also evolving. This falls under the definition of Multi-Agent Learning, as can be seen with works ...
3 votes
Accepted

Is there any value given to each chess piece in AlphaZero?

is there a value given for each piece (e.g. 1 for pawn, 3 for knight, 9 for queen, etc.) to train the algorithm, or does the algorithm learn this by himself? No, there are no such explicit values ...
  • 9,509
3 votes
Accepted

What part of the game is the value network trained to predict a winner on?

To my understanding, this is basically a supervised learning problem, where from the self play we have games associated with their winners, and the network is being trained to map game states to ...
  • 24.6k
3 votes
Accepted

Would AlphaGo Zero become perfect with enough training time?

We cannot tell with certainty whether AlphaGo Zero would become perfect with enough training time. This is because none of the parts (Neural Network) that would benefit from infinite training time (= ...
  • 9,509
3 votes
Accepted

How does AlphaZero's MCTS work when starting from the root node?

I looked at the Python pseudo-code attached to the Data S1 of the Supplementary Materials of the AlphaZero paper. Here is my findings: Contrary to the paper, AlphaZero does not store $\{N(s, a), W(S, ...
  • 1,260
3 votes
Accepted

Is it practical to train AlphaZero or MuZero (for indie games) on a personal computer?

The vast majority of neural networks are now trained on graphics processing units (GPUs) or specialised accelerator hardware such as tensor processing units (TPUs). In Mastering Chess and Shogi by ...
  • 940
3 votes

Why does Alpha Zero's Neural Network flip the board to be oriented towards the current player?

I am not an expert in RL. I have been playing Go for some years. Let's quote from AlphaZero's paper first: Aside from komi, the rules of Go are also invariant to colour transposition; this knowledge ...
  • 131
3 votes
Accepted

What is a policy training target in AlphaZero?

The training target for the policy is always exactly the one described by the pseudocode; distribution proportional to visit counts, without any other kind of scaling. The softmax sampling with the ...
  • 9,509
3 votes
Accepted

What can we learn from AlphaZero in the development towards AGI?

Some learnings from AlphaZero: Self-play, and more generally sandbox training, is effective. This indicates that given the right enviroment and enough computational power we can build highly ...
  • 373
2 votes

Why were Chess experts surprised by the AlphaZero's victory against Stockfish?

I see, based on the articles you provide, many levels of surprise in the victory: Chess is hard game to master and the counter part had the world's best practices, AlphaZero had tabula rasa. ...
  • 927
2 votes
Accepted

Alphazero policy head loss not decreasing

The loss of the policy head here is really quite different from losses in, for instance, more "conventional" Supervised Learning approaches (where we typically expect/hope to see a relatively steady ...
  • 9,509

Only top scored, non community-wiki answers of a minimum length are eligible