20
votes
Accepted
Why does the policy network in AlphaZero work?
The output of the policy network is as described in the original paper:
A move in chess may be described in two parts: selecting the piece to move, and then
selecting among the legal moves for ...
- 27k
7
votes
Did Alphago zero actually beat Alphago 100 games to 0?
Did AlphaGo and AlphaGo [Zero] play 100 repetitions of the same sequence of boards, or were there 100 different games?
There were 100 different games. You can view some example games between AlphaGo [...
- 27k
7
votes
Accepted
Why is the merged neural network of AlphaGo Zero more efficient than two separate neural networks?
Why has this merge proven beneficial?
If you think about the shared Value/Policy network as consisting of a shared component (the Residual Network layers) with a Value and Policy component on top ...
- 406
6
votes
Accepted
What is the difference between DQN and AlphaGo Zero?
DQN and AlphaZero do not share much in terms of implementation.
However, they are based on the same Reinforcement Learning (RL) theoretical framework. If you understand terms like MDP, reward, return, ...
- 27k
5
votes
Accepted
What is a "logit probability"?
Indeed I haven't seen the term "logit probability" used in many places other than that specific paper. So, I cannot really comment on why they're using that term / where it comes from / if anyone else ...
- 9,824
4
votes
Accepted
What part of the game is the value network trained to predict a winner on?
To my understanding, this is basically a supervised learning problem, where from the self play we have games associated with their winners, and the network is being trained to map game states to ...
- 27k
4
votes
Would AlphaGo Zero become perfect with enough training time?
Assuming you mean a mathematically perfect player, similar to what we can achieve trivially in Tic Tac Toe, then the answer is "maybe". The underlying reinforcement learning algorithms that it uses do ...
- 27k
4
votes
Accepted
How Does AlphaGo Zero Implement Reinforcement Learning?
If you learn a policy or a value function from experience (that is, interaction with an environment), that's RL. In the case of AlphaGo, the MCTS is used to acquire the experience.
RL could in fact ...
- 38.2k
3
votes
Accepted
Why does AlphaGo Zero select move based on exponentiated visit count?
The answer is surprisingly hidden in the original AlphaGo paper:
At the end of search AlphaGo selects the action with maximum visit count; this
is less sensitive to outliers than maximizing ...
- 1,122
3
votes
Would it take 1700 years to run AlphaGo Zero in commodity hardware?
Although the above statement holds important analogies to communicate the technical advances made by deep mind in the development of Alpha Go. It is inaccurate and should be taken skeptically.
...
- 1,176
3
votes
Accepted
Would AlphaGo Zero become perfect with enough training time?
We cannot tell with certainty whether AlphaGo Zero would become perfect with enough training time. This is because none of the parts (Neural Network) that would benefit from infinite training time (= ...
- 9,824
2
votes
Would AlphaGo Zero become perfect with enough training time?
Yes AlphaGo Zero could become undeniably perfect.
It has won 100:0 against AlphaGo Lee (which won 4:1 against 18-time world champion (human) Lee Sedol) and 89:11 against AlphaGo Master (which won 60 ...
- 616
2
votes
Why is Monte Carlo used as the tree search algorithm for AlphaGo?
The paper that introduced AlphaGo, Mastering the game of Go with deep neural networks and tree search, motivates the use of MCTS
Monte Carlo tree search (MCTS) uses Monte Carlo rollouts to estimate ...
- 38.2k
2
votes
What is the input to AlphaGo's neural network?
The input to the neural network is a $19 × 19 × 17$ image stack
comprising $17$ binary feature planes. $8$ feature planes $X_t$ consist of binary values indicating the
presence of the current ...
- 2,018
2
votes
In Alpha(Go)Zero, why is the policy extracted from MCTS better than the network one?
Basically, the policy network just looks at the current state and doesn't have any added benefit from searching. I think of the policy network as a chess player's initial candidate move selection and ...
- 125
2
votes
In Alpha(Go)Zero, why is the policy extracted from MCTS better than the network one?
The most important word for answering your question from that quote from the paper is probably the word "usually": These search probabilities usually select much stronger moves than the raw ...
- 9,824
2
votes
Does AlphaGo play random moves in a real competition?
Question 1: I don't think they ran AlphaGo or AlphaGoZero in training mode during tournament matches because the computing power required for this is really large. I don't recall if this is described ...
- 179
2
votes
How does policy network learn in AlphaZero?
I'll first address the big-picture intuition and then address each point separately.
First of all, the tree search tries to find the best policy at each turn. Losing the game doesn't necessarily mean ...
- 338
2
votes
Accepted
AlphaGo Zero: does $Q(s_t, a)$ dominate $U(s_t, a)$ in difficult game states?
I don't think you've necessarily made any real mistakes in your calculations or anything like that, that all seems accurate. I can't really confidently answer your questions about "Does X usually ...
- 9,824
2
votes
Is which sense was AlphaGo "just given a rule book"?
it was "just given the rulebook", what does this mean? Literally a book written in English to read?
The program was not given a natural language version of the rules to interpret. That ...
- 27k
1
vote
Is it possible for AlphaGo Zero to use recurrent networks to achieve similar performance?
I can't think of any reason why using an RNN wouldn't work in theory.
In practice RNNs are slightly harder to train (they can be unstable, and ever more practically you have to deal with multiple ...
- 338
1
vote
Does AlphaGo play random moves in a real competition?
AlphaGo/AlphaZero has 3 main sources of randomness during competitive mode:
Move temperature: The MCTS process outputs a probability distribution P over all candidate moves. The agent chooses a move ...
- 161
1
vote
Does AlphaGo play random moves in a real competition?
The core mechanics of AlphaZero during selfplay and real tournament games are the same: something similar to Monte Carlo Tree Search is done but guided by the current neural network instead of random ...
- 338
1
vote
Why is tree search/planning used in reinforcement learning?
Some sources say MCTS (or planning in general) increases the sample efficiency.
If we're thinking purely about experiments run in simulations, then I'd estimate there may be cases where a combination ...
- 9,824
1
vote
What is the search depth of AlphaGo and AlphaGo Zero?
For easier visualization, I recommend this video:
https://twitter.com/i/status/1257053365424578565
The more detailed article about GO algorithms:
https://deepmind.com/blog/article/alphago-zero-...
- 144
1
vote
Accepted
How does the AlphaGo Zero policy decide what move to execute?
The formula in question uses a function N(state, action) that defines a visit count of a state-action pair (introduced on page 3). To describe how it is used, lets first describe the steps of AlphaGo ...
- 3,727
1
vote
How do the achievements met in the gaming field (ex. AlphaGo Zero) impact other fields of application?
Yes it's created something important. Until Alpha(Go) Zero all (or almost all) of Deep Learning approach to Reinforcement Learning was based on Time Difference loss function. The weakness of Time ...
- 695
1
vote
What is the difference between DQN and AlphaGo Zero?
You can actually combine AlphaZero-like approach with DQN:
A* + DQN
- 695
Only top scored, non community-wiki answers of a minimum length are eligible