Questions tagged [alphazero]

For questions related to DeepMind's AlphaZero, which is a computer program that can play Go, Chess, and Shogi. AlphaZero achieved, within 24 hours of training, a superhuman level of play in these three games by defeating world-champion programs Stockfish, Elmo, and the 3-day version of AlphaGo Zero. AlphaZero was introduced in "Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm" (2017) by David Silver et al.

Filter by
Sorted by
Tagged with
10
votes
3answers
1k views

Why were Chess experts surprised by the AlphaZero's victory against Stockfish?

It was recently brought to my attention that Chess experts took the outcome of this now famous match as something of an upset. See: Chess’s New Best Player Is A Fearless, Swashbuckling Algorithm ...
10
votes
3answers
757 views

Does Monte Carlo tree search qualify as machine learning?

To the best of my understanding, the Monte Carlo tree search (MCTS) algorithm is an alternative to minimax for searching a tree of nodes. It works by choosing a move (generally, the one with the ...
10
votes
1answer
2k views

Why does the policy network in AlphaZero work?

In AlphaZero, the policy network (or head of the network) maps game states to a distribution of the likelihood of taking each action. This distribution covers all possible actions from that state. ...
8
votes
2answers
160 views

How can alpha zero learn if the tree search stops and restarts before finishing a game?

I am trying to understand how alpha zero works, but there is one point that I have problems understanding, even after reading several different explanations. As I understand it (see for example https:/...
7
votes
1answer
465 views

Does AlphaZero use Q-Learning?

I was reading the AlphaZero paper Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm, and it seems they don't mention Q-Learning anywhere. So does AZ use Q-...
6
votes
3answers
222 views

Would AlphaGo Zero become perfect with enough training time?

Would AlphaGo Zero become theoretically perfect with enough training time? If not, what would be the limiting factor? (By perfect, I mean it always wins the game if possible, even against another ...
5
votes
2answers
1k views

What is the difference between DQN and AlphaGo Zero?

I have already implemented a relatively simple DQN on Pacman. Now I would like to clearly understand the difference between a DQN and the techniques used by AlphaGo zero/AlphaZero and I couldn't find ...
4
votes
1answer
70 views

Can AlphaZero considered as Multi-Agent Deep Reinforcement Learning?

Can AlphaZero considered as Multi-Agent Deep Reinforcement Learning? I could not find a clear answer on this. I would say yes it is Multi Agent Learning, as there are two Agents playing against each ...
4
votes
1answer
72 views

How can I use one neural network for both players in Alpha Zero (Connect 4)?

First of all, it is great to have found this community! I am currently implementing my own Alpha Zero clone on Connect4. However, I have a mental barrier I cannot overcome. How can I use one neural ...
4
votes
2answers
503 views

What part of the game is the value network trained to predict a winner on?

The Alpha Zero (as well as AlphaGo Zero) papers say they trained the value head of the network by "minimizing the error between the predicted winner and the game winner" throughout its many self-play ...
4
votes
1answer
59 views

How does AlphaZero's MCTS work when starting from the root node?

From the AlphaGo Zero paper, during MCTS, statistics for each new node are initialized as such: ${N(s_L, a) = 0, W (s_L, a) = 0, Q(s_L, a) = 0, P (s_L, a) = p_a}$. The PUCT algorithm for selecting ...
4
votes
1answer
501 views

Is there any value given to each chess piece in AlphaZero? [closed]

Recently, DeepMind's AlphaZero chess algorithm did better than the prior best chess software Stockfish. I read the paper Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning ...
4
votes
1answer
81 views

In AlphaZero, which policy is saved in the dataset, and how is the move chosen?

I've been doing some research on the principles behind AlphaZero. Especially this cheat sheet (1) and this implementation (2) (in Connect 4) were very useful. Yet, I still have two important questions:...
3
votes
1answer
89 views

Knowledge required for understanding AlphaZero paper

My goal is to understand AlphaZero paper published by deepmind. I'm beginning my journey trying to get the basic intuition of reinforcement learning from the book by Barto and Sutton. As per my ...
3
votes
1answer
132 views

Alpha zero before move 8

The Alpha zero paper says that the The first set of features are repeated for each position in a T = 8-step history. So what happens before the first 8 moves? Do they just repeat the starting position?...
3
votes
1answer
64 views

AlphaGo Zero: does $Q(s_t, a)$ dominate $U(s_t, a)$ in difficult game states?

AlphaGo Zero AlphaGo Zero uses a Monte-Carlo Tree Search where the selection phase is governed by $\operatorname*{argmax}\limits_a\left( Q(s_t, a) + U(s_t, a) \right)$, where: the exploitation ...
3
votes
1answer
92 views

When does AlphaZero play suboptimal moves?

If AlphaZero was always playing the best moves it would just generate the same training game over and over again. So where does the randomness come from? When does it decide not to play the most ...
3
votes
1answer
326 views

Alphazero policy head loss not decreasing

I am now working on training an alphazero player for a board game. The implementation of board game is mine, MCTS for alphazero was taken elsewhere. Due to complexity of the game, it takes a much ...
3
votes
0answers
41 views

Where does reinforcement learning actually show up in Deepmind's game engines?

From the brief research I've done on the topic, it appears that the way Deepmind's Alphazero or Muzero makes decisions is through Monte Carlo tree searches, where in the randomized simulations allows ...
3
votes
0answers
46 views

Is Monte Carlo tree search needed in partially observable environments during gameplay?

I understand that with a fully observable environment (chess / go etc) you can run an MCTS with an optimal policy network for future planning purposes. This will allow you to pick actions for gameplay,...
3
votes
0answers
151 views

How is the rollout from the MCTS implemented in both of the AlphaGo Zero and the AlphaZero algorithms?

In a vanilla Monte Carlo tree search (MCTS) implementation, the rollout is usually implemented following a uniform random policy, that is, it takes random actions until the game is finished and only ...
2
votes
1answer
70 views

What does it mean for AlphaZero's network to be “fully trained”

Reading this blog post about AlphaZero: https://deepmind.com/blog/article/alphazero-shedding-new-light-grand-games-chess-shogi-and-go It uses language such as "the amount of training the network ...
2
votes
2answers
74 views

What happens when an opponent a neural network is playing with does not obey the rules of the game (i.e. cheats)?

For example, if AlphaZero plays with an opponent who has a right to move chess figures any way she wants, or make more than 1 move in a turn? Will a neural network adapt to that, as it adapted to an ...
2
votes
1answer
138 views

Building 'evaluation' neural networks for go, reversi, checkers etc, how to train?

I'm trying to build neural networks for games like Go, Reversi, Othello, Checkers, or even tic-tac-toe, not by calculating a move, but by making them evaluate a position. The input is any board ...
2
votes
1answer
66 views

Similarities and differences between UCT algorithms in (i), (ii), (iii) and (iv)?

I am trying to understand the similarities and differences between: (i) the UCT algorithm in Kocsis and Szepesvári (2006); (ii) the UCT algorithm in Section 3.3 of Browne et al (2012); (iii) the MCTS ...
2
votes
0answers
10 views

Do alphazero/muzero learn faster in terms of number of games played than humans?

My apologies if this is too naive a question. I don't know much about AI and am just curious. From what I read, alphazero/muzero outperform any human chess player after a few hours of training. I have ...
2
votes
0answers
26 views

Are inputs into AlphaZero the same during the evaluate step in MCTS and during test time?

From the AlphaZero paper: The input to the neural network is an N × N × (M T + L) image stack that represents state using a concatenation of T sets of M planes of size N × N . Each set of planes ...
2
votes
1answer
42 views

In AlphaZero, do we need to store the data of terminal states?

I have a question about the training data used during the update/back-propagation step of the neural network in AlphaZero. From the paper: The data for each time-step $t$ is stored as ($s_t, \pi_t, ...
2
votes
0answers
56 views

What kind of policy evaluation and policy improvement AlphaGo, AlphaGo Zero and AlphaZero are using

I'm trying to find out what kind of policy improvement and policy evaluation AlphaGo, AlphaGo Zero, and AlphaZero are using. By looking into their respective paper and SI, I can conclude that it is a ...
2
votes
0answers
49 views

Alphazero Value loss doesn't decrease

Currenly I'm trying to reimplement alphazero in pure c++ using libtorch to accomodate my project's need. But when I training my model, I found out that the value loss doesn't decrese at all after even ...
1
vote
1answer
51 views

Alpha Zero queen promotion

"The final 9 planes encode possible underpromotions for pawn moves or captures in two possible diagonals, to knight, bishop or rook respectively. Other pawn moves or captures from the ...
1
vote
1answer
44 views

Is which sense was AlphaGo “just given a rule book”?

I was told that AlphaGo (or some related program) was not explicitly taught even the rules of Go -- if it was "just given the rulebook", what does this mean? Literally, a book written in ...
1
vote
1answer
31 views

Is there a training data capacity limit for AlphaZero (Chess)?

In AlphaZero, we collect ($s_t, \pi_t, z_t$) tuples from self-play, where $s_t$ is the board state, $\pi_t$ is the policy, and $z_t$ is the reward from winning/losing the game. In other DeepRL off-...
1
vote
1answer
70 views

In Alpha(Go)Zero, why is the policy extracted from MCTS better than the network one?

I've read through the Alpha(Go)Zero paper and there is only one thing I don't understand. The paper on page 1 states: The MCTS search outputs probabilities π of playing each move. These search ...
1
vote
3answers
513 views

Combining deep reinforcement learning with alpha-beta pruning

I will explain my question in relation to chess, but it should be relevant for other games as well: In short terms: Is it possible to combine the techniques used by AlphaZero with those used by, say, ...
1
vote
1answer
469 views

What was the average decision speed pf Alpha Zero in the recent Stockfish match?

The match got a lot of press, and I doubt anyone is surprised that Alpha Zero crushed Stockfish. See: AlphaZero Destroys Stockfish in 100 Game Match To me, what's really salient is that "much like ...
1
vote
0answers
39 views

Stack of Planes as the Action Space Representation for AlphaZero (Chess)

I have a question regarding the action space of the policy network used in AlphaZero. From the paper: We represent the policy π(a|s) by a 8 × 8 × 73 stack of planes encoding a probability ...
1
vote
0answers
33 views

How to encode board before input into the neural net?

Currently I'm working on an educational project (implementation of AlphaZero approach to different types of board games). My biggest concern at the moment is how to encode board before input into the ...
1
vote
0answers
26 views

AlphaZero value at root node not being affected by training

I have written my own AlphaZero implementation and started training it recently. Problem is, I am 99% sure there is a mistake and I do not know how to tackle this, since I cannot explain it. I am new ...
1
vote
1answer
89 views

How to deal with invalid output in a policy network? [duplicate]

I am interested in creating a neural network-based engine for chess. It uses a $8 \times 8 \times 73$ output space for each possible move as proposed in the Alpha Zero paper: Mastering Chess and Shogi ...
0
votes
1answer
571 views

How does AlphaZero use its value and policy heads in conjunction?

I have a question about how the value and policy heads are used in AlphaZero (not Alphago Zero), and where the leaf nodes are relative to the root node. Specifically, there seem to be several possible ...
0
votes
0answers
29 views

Alpha Zero does not converge for Connect 6, a game with huge branching factor - why?

I have a problem with applying alpha zero self-play to a game (Connect 6) with a huge branching factor (30,000 on average). I have implemented the MCTS as described but I found that during the MCTS ...
0
votes
0answers
32 views

In AlphaZero, which features are one-hot encoded and which are single real-valued?

From the AlphaZero paper, the caption of Table S1 (p. 13) Table S1: Input features used by AlphaZero in Go, Chess and Shogi respectively. The first set of features are repeated for each position in a ...
0
votes
0answers
66 views

What would be the AlphaGo's performance in continuous action space?

During my research for Google DeepMind's Go-playing program Alpha Go and its successor Alpha Go Zero, I discovered that the system uses a clever pipeline and an interplay of blocks of both policy and ...
0
votes
0answers
56 views

Total loss increasing, but loss components are decreasing?

I am writing a AlphaGo Zero clone, and sometimes in the training the policy head loss and value head loss would both be decreasing, but the total loss is increasing? How is this possible? I am using ...
-1
votes
0answers
19 views

Is there a treatise that contains the following formulas? [closed]

Is there a treatise that contains the following formulas? ark_value = w/n + cpuct * p * √t/(1+n) It didn't exist in alphazero's treatise, so please let me know if ...
-1
votes
1answer
77 views

What's missing from Alpha Zero to make it generally intelligent?

Alpha Zero is the game playing AI. One might try to use the algorithm in a robot which takes input from it's environment and thinks about taking the best course of action. So I was thinking about ...