20
votes
Accepted
Why does the policy network in AlphaZero work?
The output of the policy network is as described in the original paper:
A move in chess may be described in two parts: selecting the piece to move, and then
selecting among the legal moves for ...
- 30.2k
17
votes
Accepted
Why were Chess experts surprised by the AlphaZero's victory against Stockfish?
Good question.
First and foremost is that in Go deepmind had no superhuman opponents to challenge. Go engines were not anywhere near the highest level of the top human players. In chess, however, ...
- 186
14
votes
Accepted
Is AlphaZero an example of an AGI?
Good question!
AlphaZero, though a major milestone, is most definitely not an AGI :)
AlphaGo, though strong at the game of Go, is narrowly strong ("strong-narrow AI"), defined as strength in a ...
- 6,237
10
votes
Does Monte Carlo tree search qualify as machine learning?
John's answer is correct in that MCTS is traditionally not viewed as a Machine Learning approach, but as a tree search algorithm, and that AlphaZero combines this with Machine Learning techniques (...
- 10.1k
9
votes
Accepted
Does Monte Carlo tree search qualify as machine learning?
Monte Carlo Tree Search is not usually thought of as a machine learning technique, but as a search technique. There are parallels (MCTS does try to learn general patterns from data, in a sense, but ...
- 9,207
7
votes
Accepted
Does AlphaZero use Q-Learning?
Note: you mentioned in the comments that you are reading the old, pre-print version of the paper describing AlphaZero on arXiv. My answer will be for the "official", peer-reviewed, more recent ...
- 10.1k
6
votes
Accepted
What is the difference between DQN and AlphaGo Zero?
DQN and AlphaZero do not share much in terms of implementation.
However, they are based on the same Reinforcement Learning (RL) theoretical framework. If you understand terms like MDP, reward, return, ...
- 30.2k
6
votes
Accepted
Clarifying representation of Neural Nerwork input for Chess Alpha Zero
For anyone wondering, I believe to have found the answer:
Yes, it will be an 8x8 plane where all the entries are the same, the number of moves (or mpves with no progress).
There are two repetitions ...
- 76
6
votes
Accepted
How does the Alpha Zero's move encoding work?
Let's do the code, so all the details are down.
Encoding dictionary:
...
- 2,426
5
votes
Accepted
How does AlphaZero use its value and policy heads in conjunction?
I am also a bit confused by your wording but I will try to clear some things up.
During MCTS the policy head is used to guide the search while the value head is used as a replacement for roll outs to ...
- 66
5
votes
Accepted
Does the AlphaZero algorithm keep the subtree statistics after each move during MCTS?
The supplementary material of the AlphaZero paper states the following:
Unless
otherwise specified, the training and search algorithm and parameters are identical to AlphaGo
Zero.
I didn't see any ...
- 1,282
4
votes
Why were Chess experts surprised by the AlphaZero's victory against Stockfish?
MCTS for chess had been tried in the literature with little success. It was assumed AlphaGo's approach would never work on chess, maybe in Go but not in chess. Suddenly, Google announced the approach ...
- 1,411
4
votes
Accepted
What part of the game is the value network trained to predict a winner on?
To my understanding, this is basically a supervised learning problem, where from the self play we have games associated with their winners, and the network is being trained to map game states to ...
- 30.2k
4
votes
Would AlphaGo Zero become perfect with enough training time?
Assuming you mean a mathematically perfect player, similar to what we can achieve trivially in Tic Tac Toe, then the answer is "maybe". The underlying reinforcement learning algorithms that it uses do ...
- 30.2k
4
votes
Accepted
What happens before the first 8 moves in Alpha Zero?
On page 13, right under Table S1 in the linked paper, this is explained (emphasis in bold at the end mine):
Each set of planes represents the board position at a time-step $t - T + 1, \dots, t$, ...
- 10.1k
4
votes
Accepted
What knowledge is required for understanding the AlphaZero paper?
The more you read, the more deeply you can understand any paper, but given your stated background, reading the Monte-Carlo Tree Search chapter of Barto & Sutton, plus Gerald Tesauro's TD-Gammon ...
- 9,207
4
votes
Accepted
How does AlphaZero's MCTS work when starting from the root node?
I looked at the Python pseudo-code attached to the Data S1 of the Supplementary Materials of the AlphaZero paper. Here is my findings:
Contrary to the paper, AlphaZero does not store $\{N(s, a), W(S, ...
- 1,510
4
votes
Accepted
What can we learn from AlphaZero in the development towards AGI?
Some learnings from AlphaZero:
Self-play, and more generally sandbox training, is effective. This indicates that given the right enviroment and enough computational power we can build highly ...
- 1,141
3
votes
Accepted
Can AlphaZero considered as Multi-Agent Deep Reinforcement Learning?
Depends on perspective.
On one hand, you have an agent playing in an environment with another agent also evolving. This falls under the definition of Multi-Agent Learning, as can be seen with works ...
- 906
3
votes
Accepted
When does AlphaZero play suboptimal moves?
During the self-play training process, AlphaZero does not greedily play only the moves it thinks are "best" (which would normally be the move with the highest visit count leading out of the root node ...
- 10.1k
3
votes
Accepted
What does it mean for AlphaZero's network to be "fully trained"
Neural network will eventually reach limit of it's approximation power. You can't expect to learn more and more things infinitely long with the same amount of learnable parameters. Also, if you ...
- 2,336
3
votes
Alpha Zero queen promotion
It means that there is no explicit coding of action choices to promote to queen, it is the default assumption if the underpromotion actions are not taken.
The Alpha Zero chess implementation can ...
- 30.2k
3
votes
Accepted
How can I use one neural network for both players in Alpha Zero (Connect 4)?
Let's define your problem from another point of view. Let's say that in this RL problem you have two agents (agent1 and agent2) that compete with each other in order to accomplish their own goal, i.e.,...
- 46
3
votes
Accepted
Is there any value given to each chess piece in AlphaZero?
is there a value given for each piece (e.g. 1 for pawn, 3 for knight, 9 for queen, etc.) to train the algorithm, or does the algorithm learn this by himself?
No, there are no such explicit values ...
- 10.1k
3
votes
Combining deep reinforcement learning with alpha-beta pruning
Yes it's possible to to combine AlphaZero with Minimax methods (including alpha-beta pruning). AlphaZero itself is combination of Monte Carlo Tree Search (MCTS) and Deep Network, where MCTS is used ...
- 705
3
votes
Accepted
Would AlphaGo Zero become perfect with enough training time?
We cannot tell with certainty whether AlphaGo Zero would become perfect with enough training time. This is because none of the parts (Neural Network) that would benefit from infinite training time (= ...
- 10.1k
3
votes
Accepted
Is it practical to train AlphaZero or MuZero (for indie games) on a personal computer?
The vast majority of neural networks are now trained on graphics processing units (GPUs) or specialised accelerator hardware such as tensor processing units (TPUs).
In Mastering Chess and Shogi by ...
- 1,000
3
votes
Why does Alpha Zero's Neural Network flip the board to be oriented towards the current player?
I am not an expert in RL. I have been playing Go for some years.
Let's quote from AlphaZero's paper first:
Aside from
komi, the rules of Go are also invariant to colour transposition; this knowledge ...
- 131
3
votes
Accepted
At what point are MCTS results discarded in AlphaZero Training?
You're right, it would not be great to keep training on games played early during the selfplay process, since they would just hurt the network playing strength.
The AlphaZero paper (free access) ...
- 373
3
votes
Accepted
What is a policy training target in AlphaZero?
The training target for the policy is always exactly the one described by the pseudocode; distribution proportional to visit counts, without any other kind of scaling.
The softmax sampling with the ...
- 10.1k
Only top scored, non community-wiki answers of a minimum length are eligible