18 votes
Accepted

Why does the policy network in AlphaZero work?

The output of the policy network is as described in the original paper: A move in chess may be described in two parts: selecting the piece to move, and then selecting among the legal moves for ...
user avatar
  • 23.1k
16 votes
Accepted

Why were Chess experts surprised by the AlphaZero's victory against Stockfish?

Good question. First and foremost is that in Go deepmind had no superhuman opponents to challenge. Go engines were not anywhere near the highest level of the top human players. In chess, however, ...
user avatar
  • 176
14 votes
Accepted

Is AlphaZero an example of an AGI?

Good question! AlphaZero, though a major milestone, is most definitely not an AGI :) AlphaGo, though strong at the game of Go, is narrowly strong ("strong-narrow AI"), defined as strength in a ...
user avatar
  • 6,077
9 votes

Does Monte Carlo tree search qualify as machine learning?

John's answer is correct in that MCTS is traditionally not viewed as a Machine Learning approach, but as a tree search algorithm, and that AlphaZero combines this with Machine Learning techniques (...
user avatar
  • 9,316
9 votes
Accepted

Does Monte Carlo tree search qualify as machine learning?

Monte Carlo Tree Search is not usually thought of as a machine learning technique, but as a search technique. There are parallels (MCTS does try to learn general patterns from data, in a sense, but ...
user avatar
7 votes
Accepted

Does AlphaZero use Q-Learning?

Note: you mentioned in the comments that you are reading the old, pre-print version of the paper describing AlphaZero on arXiv. My answer will be for the "official", peer-reviewed, more recent ...
user avatar
  • 9,316
6 votes
Accepted

What is the difference between DQN and AlphaGo Zero?

DQN and AlphaZero do not share much in terms of implementation. However, they are based on the same Reinforcement Learning (RL) theoretical framework. If you understand terms like MDP, reward, return, ...
user avatar
  • 23.1k
6 votes
Accepted

Clarifying representation of Neural Nerwork input for Chess Alpha Zero

For anyone wondering, I believe to have found the answer: Yes, it will be an 8x8 plane where all the entries are the same, the number of moves (or mpves with no progress). There are two repetitions ...
user avatar
4 votes

Why were Chess experts surprised by the AlphaZero's victory against Stockfish?

MCTS for chess had been tried in the literature with little success. It was assumed AlphaGo's approach would never work on chess, maybe in Go but not in chess. Suddenly, Google announced the approach ...
user avatar
  • 1,368
4 votes

Would AlphaGo Zero become perfect with enough training time?

Assuming you mean a mathematically perfect player, similar to what we can achieve trivially in Tic Tac Toe, then the answer is "maybe". The underlying reinforcement learning algorithms that it uses do ...
user avatar
  • 23.1k
4 votes
Accepted

What happens before the first 8 moves in Alpha Zero?

On page 13, right under Table S1 in the linked paper, this is explained (emphasis in bold at the end mine): Each set of planes represents the board position at a time-step $t - T + 1, \dots, t$, ...
user avatar
  • 9,316
4 votes
Accepted

What knowledge is required for understanding the AlphaZero paper?

The more you read, the more deeply you can understand any paper, but given your stated background, reading the Monte-Carlo Tree Search chapter of Barto & Sutton, plus Gerald Tesauro's TD-Gammon ...
user avatar
4 votes
Accepted

How does the Alpha Zero's move encoding work?

Let's do the code, so all the details are down. Encoding dictionary: ...
user avatar
  • 1,763
3 votes
Accepted

What does it mean for AlphaZero's network to be "fully trained"

Neural network will eventually reach limit of it's approximation power. You can't expect to learn more and more things infinitely long with the same amount of learnable parameters. Also, if you ...
user avatar
  • 2,216
3 votes

Alpha Zero queen promotion

It means that there is no explicit coding of action choices to promote to queen, it is the default assumption if the underpromotion actions are not taken. The Alpha Zero chess implementation can ...
user avatar
  • 23.1k
3 votes
Accepted

How can I use one neural network for both players in Alpha Zero (Connect 4)?

Let's define your problem from another point of view. Let's say that in this RL problem you have two agents (agent1 and agent2) that compete with each other in order to accomplish their own goal, i.e.,...
user avatar
  • 46
3 votes
Accepted

Can AlphaZero considered as Multi-Agent Deep Reinforcement Learning?

Depends on perspective. On one hand, you have an agent playing in an environment with another agent also evolving. This falls under the definition of Multi-Agent Learning, as can be seen with works ...
user avatar
3 votes
Accepted

How does AlphaZero use its value and policy heads in conjunction?

I am also a bit confused by your wording but I will try to clear some things up. During MCTS the policy head is used to guide the search while the value head is used as a replacement for roll outs to ...
user avatar
  • 46
3 votes
Accepted

Is there any value given to each chess piece in AlphaZero?

is there a value given for each piece (e.g. 1 for pawn, 3 for knight, 9 for queen, etc.) to train the algorithm, or does the algorithm learn this by himself? No, there are no such explicit values ...
user avatar
  • 9,316
3 votes
Accepted

What part of the game is the value network trained to predict a winner on?

To my understanding, this is basically a supervised learning problem, where from the self play we have games associated with their winners, and the network is being trained to map game states to ...
user avatar
  • 23.1k
3 votes
Accepted

Would AlphaGo Zero become perfect with enough training time?

We cannot tell with certainty whether AlphaGo Zero would become perfect with enough training time. This is because none of the parts (Neural Network) that would benefit from infinite training time (= ...
user avatar
  • 9,316
3 votes
Accepted

Is it practical to train AlphaZero or MuZero (for indie games) on a personal computer?

The vast majority of neural networks are now trained on graphics processing units (GPUs) or specialised accelerator hardware such as tensor processing units (TPUs). In Mastering Chess and Shogi by ...
user avatar
  • 931
3 votes
Accepted

What is a policy training target in AlphaZero?

The training target for the policy is always exactly the one described by the pseudocode; distribution proportional to visit counts, without any other kind of scaling. The softmax sampling with the ...
user avatar
  • 9,316
2 votes

Why were Chess experts surprised by the AlphaZero's victory against Stockfish?

I see, based on the articles you provide, many levels of surprise in the victory: Chess is hard game to master and the counter part had the world's best practices, AlphaZero had tabula rasa. ...
user avatar
  • 927
2 votes
Accepted

Alphazero policy head loss not decreasing

The loss of the policy head here is really quite different from losses in, for instance, more "conventional" Supervised Learning approaches (where we typically expect/hope to see a relatively steady ...
user avatar
  • 9,316
2 votes
Accepted

How can alpha zero learn if the tree search stops and restarts before finishing a game?

You're right that AlphaGo Zero doesn't perform rollouts in its MCTS. It does complete many, many games, though. Realize that AlphaGo Zero only iterates MCTS 1,600 times before taking an action. The ...
user avatar
2 votes

Combining deep reinforcement learning with alpha-beta pruning

Yes it's possible to to combine AlphaZero with Minimax methods (including alpha-beta pruning). AlphaZero itself is combination of Monte Carlo Tree Search (MCTS) and Deep Network, where MCTS is used ...
user avatar
2 votes
Accepted

When does AlphaZero play suboptimal moves?

During the self-play training process, AlphaZero does not greedily play only the moves it thinks are "best" (which would normally be the move with the highest visit count leading out of the root node ...
user avatar
  • 9,316
2 votes

What happens when an opponent a neural network is playing with does not obey the rules of the game (i.e. cheats)?

The behaviour when playing against "cheats" depends on how the agent has been trained, and how different the game becomes from the training scenarios. It will also depend on how much of the ...
user avatar
  • 23.1k
2 votes

Is which sense was AlphaGo "just given a rule book"?

it was "just given the rulebook", what does this mean? Literally a book written in English to read? The program was not given a natural language version of the rules to interpret. That ...
user avatar
  • 23.1k

Only top scored, non community-wiki answers of a minimum length are eligible