Questions tagged [alphago-zero]
For questions related to AlphaGo Zero, which is a version of DeepMind's Go software, AlphaGo, that does not use data from human games and it is stronger than AlphaGo. There is a generalized version of AlphaGo Zero called AlphaZero, which beat the 3-day version of AlphaGo Zero by winning 60 games to 40. AlphaGo Zero was introduced in the paper "Mastering the game of Go without human knowledge" (2017) by David Silver et al.
31
questions
1
vote
2
answers
21
views
How does Alpha Go Zero MCTS work in parallel?
I am trying to better understand the article "Mastering the Game of Go without Human Knowledge" (link) and I'm confused about the parallel implementation of Monte-Carlo-Tree-Search.
On page ...
- 245
1
vote
1
answer
74
views
What do simulations mean in the context of the AlphaGoZero paper?
I am familiar with the Monte Carlo Tree Search, which can be broken down into 4 parts - Selection, Expansion, Simulation and Backpropogation. In this case, the simulation step denotes performing a ...
- 588
1
vote
1
answer
61
views
When do we use the neural network to predict value during the expansion stage of MCTS in the AlphaZero algorithm?
According to what I understand from the AlphaZero algorithm, a neural network is used to set value and prior probability for a node during the expansion stage of MCTS. On the other hand, according to ...
1
vote
1
answer
91
views
Is it possible for AlphaGo Zero to use recurrent networks to achieve similar performance?
AlphaGo Zero stacks 7 board history along with the current board together to form the input to the network. However, is it possible to use an RNN to replace the input of history and achieve similar ...
- 203
2
votes
3
answers
372
views
Does AlphaGo play random moves in a real competition?
Alphago and AlphaGo zero use random play to generate data and use the data to train DNN. "Random play" means that there is a positive probability for AlphaGo to play some suboptimal moves ...
- 163
1
vote
0
answers
40
views
Is there a way to beat AlphaGo Zero with different method?
As I read the research from
https://deepmind.com/research
It seem AlphagoZero use zero knowledge and use Reinforcement learning to improve the ai skill of playing.
Is there a way to beat AlphagoZero? ...
- 175
4
votes
1
answer
529
views
How does policy network learn in AlphaZero?
I'm currently trying to understand how AlphaZero works. There is one thing with the training of the AlphaZero's policy head that confuses me. Basically, in AlphaGo Zero's paper (where the major part ...
- 41
3
votes
1
answer
206
views
AlphaGo Zero: does $Q(s_t, a)$ dominate $U(s_t, a)$ in difficult game states?
AlphaGo Zero
AlphaGo Zero uses a Monte-Carlo Tree Search where the selection phase is governed by $\operatorname*{argmax}\limits_a\left( Q(s_t, a) + U(s_t, a) \right)$, where:
the exploitation ...
- 1,510
1
vote
1
answer
66
views
Is which sense was AlphaGo "just given a rule book"?
I was told that AlphaGo (or some related program) was not explicitly taught even the rules of Go -- if it was "just given the rulebook", what does this mean? Literally, a book written in ...
- 141
2
votes
1
answer
485
views
Did Alphago zero actually beat Alphago 100 games to 0?
tl;dr
Did AlphaGo and AlphaGo play 100 repetitions of the same sequence of boards, or were there 100 different games?
Background:
Alphago was the first superhuman go player, but it had human tuning ...
- 361
2
votes
1
answer
389
views
Why is tree search/planning used in reinforcement learning?
In AlphaGo Zero, MCTS is used along with policy networks. Some sources say MCTS (or planning in general) increases the sample efficiency.
Assumed the transition model is known and the computational ...
- 153
0
votes
1
answer
316
views
What is the search depth of AlphaGo and AlphaGo Zero?
I cannot find reliable sources but someone says it is 40 moves and someone else says it is 50+ moves. I read their papers and they use value function (NN) and policy function to trim the tree, so more ...
- 163
1
vote
2
answers
440
views
In Alpha(Go)Zero, why is the policy extracted from MCTS better than the network one?
I've read through the Alpha(Go)Zero paper and there is only one thing I don't understand.
The paper on page 1 states:
The MCTS search outputs probabilities π of playing each move. These search ...
- 43
1
vote
1
answer
100
views
How AlphaGo Zero is learning from $\pi_t$ when $z_t = -1$?
I have questions on the way AlphaGo Zero is trained.
From original AlphaGo Zero paper, I knew that AlphaGo Zero agent learns a policy, value functions by the gathered data $\{(s_t, \pi_t, z_t)\}$ ...
- 13
3
votes
1
answer
245
views
Why does AlphaGo Zero select move based on exponentiated visit count?
From the AlphaGo Zero paper, AlphaGo Zero uses an exponentiated visit count from the tree search.
Why use visit count instead of the mean action value $Q(s, a)$?
- 133
3
votes
2
answers
204
views
How does the AlphaGo Zero policy decide what move to execute?
I was going through the AlphaGo Zero paper and I was trying to understand everything, but I just can't figure out this one formula:
$$
\pi(a \mid s_0) = \frac{N(s_0, a)^{\frac{1}{\tau}}}{\sum_b N(s_0,...
- 33
0
votes
0
answers
46
views
How does (or should) AlphaGoZero (which does chess) fare against Deep Blue?
Deep blue is good at chess, but is more "hand-coded" or "top-down".
https://en.wikipedia.org/wiki/Deep_Blue_(chess_computer)
AlphaGoZero is "self-taught", and at Go is very much super-human.
https://...
- 361
2
votes
0
answers
274
views
AlphaGo Zero: Does the policy head give a probability for every possible move?
If I understood correctly, the AlphaGo Zero network returns two values: a vector of logit probabilities p and a value v.
My question is: in this vector that it is outputted, do we have a probability ...
- 255
6
votes
0
answers
1k
views
How is the rollout from the MCTS implemented in both of the AlphaGo Zero and the AlphaZero algorithms?
In the vanilla Monte Carlo tree search (MCTS) implementation, the rollout is usually implemented following a uniform random policy, that is, it takes random actions until the game is finished and only ...
- 255
2
votes
0
answers
99
views
Alphazero Value loss doesn't decrease
Currenly I'm trying to reimplement alphazero in pure c++ using libtorch to accomodate my project's need. But when I training my model, I found out that the value loss doesn't decrese at all after even ...
- 21
4
votes
1
answer
324
views
How Does AlphaGo Zero Implement Reinforcement Learning?
AlphaGo Zero (https://deepmind.com/blog/alphago-zero-learning-scratch/) has several key components that contribute to it's success:
A Monte Carlo Tree Search Algorithm that allows it to better search ...
- 521
3
votes
1
answer
554
views
What is the input to AlphaGo's neural network?
I have been reading an article on AlphaGo and one sentence confused me a little bit, because I'm not sure what it exactly means. The article says:
AlphaGo Zero only uses the black and white stones ...
user24093
2
votes
1
answer
37
views
How do the achievements met in the gaming field (ex. AlphaGo Zero) impact other fields of application?
How can we use the ability of AlphaGo Zero computer, to do something in any other life important related field? Is it possible to make something important besides having created something so smart ...
- 343
3
votes
1
answer
235
views
Why is Monte Carlo used as the tree search algorithm for AlphaGo?
Could a better algorithm other than Monte Carlo be used for the AlphaGo computer? Why didn't the DeepMind team think of choosing another kind of algorithm rather than spending time on their neural ...
- 343
5
votes
2
answers
3k
views
What is the difference between DQN and AlphaGo Zero?
I have already implemented a relatively simple DQN on Pacman.
Now I would like to clearly understand the difference between a DQN and the techniques used by AlphaGo zero/AlphaZero and I couldn't find ...
- 153
5
votes
1
answer
1k
views
What is a "logit probability"?
DeepMind's paper "Mastering the game of Go without human knowledge" states in its "Methods" section on its "Neural network architecture" that the output layer of AlphaGo Zero's policy head is "A fully ...
- 153
15
votes
1
answer
3k
views
Why does the policy network in AlphaZero work?
In AlphaZero, the policy network (or head of the network) maps game states to a distribution of the likelihood of taking each action. This distribution covers all possible actions from that state.
...
- 2,225
5
votes
2
answers
667
views
What part of the game is the value network trained to predict a winner on?
The Alpha Zero (as well as AlphaGo Zero) papers say they trained the value head of the network by "minimizing the error between the predicted winner and the game winner" throughout its many self-play ...
- 2,225
7
votes
3
answers
327
views
Would AlphaGo Zero become perfect with enough training time?
Would AlphaGo Zero become theoretically perfect with enough training time? If not, what would be the limiting factor?
(By perfect, I mean it always wins the game if possible, even against another ...
- 261
4
votes
1
answer
1k
views
Would it take 1700 years to run AlphaGo Zero in commodity hardware?
From this link, AlphaGo would take millennia to run in regular hardware.
They generated 29 million games for the final result, which means it's going to take me about 1700 years to replicate this.
...
- 906
11
votes
1
answer
1k
views
Why is the merged neural network of AlphaGo Zero more efficient than two separate neural networks?
AlphaGo Zero contains several improvements compared to its predecessors. Architectural details of Alpha Go Zero can be seen in this cheat sheet.
One of those improvements is using a single neural ...
- 1,684