20 votes
Accepted

Why does the policy network in AlphaZero work?

The output of the policy network is as described in the original paper: A move in chess may be described in two parts: selecting the piece to move, and then selecting among the legal moves for ...
Neil Slater's user avatar
  • 31.5k
7 votes

Did Alphago zero actually beat Alphago 100 games to 0?

Did AlphaGo and AlphaGo [Zero] play 100 repetitions of the same sequence of boards, or were there 100 different games? There were 100 different games. You can view some example games between AlphaGo [...
Neil Slater's user avatar
  • 31.5k
7 votes
Accepted

Why is the merged neural network of AlphaGo Zero more efficient than two separate neural networks?

Why has this merge proven beneficial? If you think about the shared Value/Policy network as consisting of a shared component (the Residual Network layers) with a Value and Policy component on top ...
mjul's user avatar
  • 416
6 votes
Accepted

What is the difference between DQN and AlphaGo Zero?

DQN and AlphaZero do not share much in terms of implementation. However, they are based on the same Reinforcement Learning (RL) theoretical framework. If you understand terms like MDP, reward, return, ...
Neil Slater's user avatar
  • 31.5k
6 votes
Accepted

What is a "logit probability"?

Indeed I haven't seen the term "logit probability" used in many places other than that specific paper. So, I cannot really comment on why they're using that term / where it comes from / if anyone else ...
Dennis Soemers's user avatar
  • 10.2k
4 votes
Accepted

What part of the game is the value network trained to predict a winner on?

To my understanding, this is basically a supervised learning problem, where from the self play we have games associated with their winners, and the network is being trained to map game states to ...
Neil Slater's user avatar
  • 31.5k
4 votes

Would AlphaGo Zero become perfect with enough training time?

Assuming you mean a mathematically perfect player, similar to what we can achieve trivially in Tic Tac Toe, then the answer is "maybe". The underlying reinforcement learning algorithms that it uses do ...
Neil Slater's user avatar
  • 31.5k
4 votes
Accepted

How Does AlphaGo Zero Implement Reinforcement Learning?

If you learn a policy or a value function from experience (that is, interaction with an environment), that's RL. In the case of AlphaGo, the MCTS is used to acquire the experience. RL could in fact ...
nbro's user avatar
  • 40.1k
3 votes
Accepted

Why does AlphaGo Zero select move based on exponentiated visit count?

The answer is surprisingly hidden in the original AlphaGo paper: At the end of search AlphaGo selects the action with maximum visit count; this is less sensitive to outliers than maximizing ...
DeepQZero's user avatar
  • 1,387
3 votes

Would it take 1700 years to run AlphaGo Zero in commodity hardware?

Although the above statement holds important analogies to communicate the technical advances made by deep mind in the development of Alpha Go. It is inaccurate and should be taken skeptically. ...
Seth Simba's user avatar
  • 1,186
3 votes
Accepted

Would AlphaGo Zero become perfect with enough training time?

We cannot tell with certainty whether AlphaGo Zero would become perfect with enough training time. This is because none of the parts (Neural Network) that would benefit from infinite training time (= ...
Dennis Soemers's user avatar
  • 10.2k
2 votes

Would AlphaGo Zero become perfect with enough training time?

Yes AlphaGo Zero could become undeniably perfect. It has won 100:0 against AlphaGo Lee (which won 4:1 against 18-time world champion (human) Lee Sedol) and 89:11 against AlphaGo Master (which won 60 ...
Rob's user avatar
  • 632
2 votes

Why is Monte Carlo used as the tree search algorithm for AlphaGo?

The paper that introduced AlphaGo, Mastering the game of Go with deep neural networks and tree search, motivates the use of MCTS Monte Carlo tree search (MCTS) uses Monte Carlo rollouts to estimate ...
nbro's user avatar
  • 40.1k
2 votes

What is the input to AlphaGo's neural network?

The input to the neural network is a $19 × 19 × 17$ image stack comprising $17$ binary feature planes. $8$ feature planes $X_t$ consist of binary values indicating the presence of the current ...
Philip Raeisghasem's user avatar
2 votes

In Alpha(Go)Zero, why is the policy extracted from MCTS better than the network one?

Basically, the policy network just looks at the current state and doesn't have any added benefit from searching. I think of the policy network as a chess player's initial candidate move selection and ...
JacKeown's user avatar
  • 125
2 votes

In Alpha(Go)Zero, why is the policy extracted from MCTS better than the network one?

The most important word for answering your question from that quote from the paper is probably the word "usually": These search probabilities usually select much stronger moves than the raw ...
Dennis Soemers's user avatar
  • 10.2k
2 votes

Does AlphaGo play random moves in a real competition?

Question 1: I don't think they ran AlphaGo or AlphaGoZero in training mode during tournament matches because the computing power required for this is really large. I don't recall if this is described ...
Lars's user avatar
  • 189
2 votes

How does policy network learn in AlphaZero?

I'll first address the big-picture intuition and then address each point separately. First of all, the tree search tries to find the best policy at each turn. Losing the game doesn't necessarily mean ...
Todd Sewell's user avatar
2 votes
Accepted

AlphaGo Zero: does $Q(s_t, a)$ dominate $U(s_t, a)$ in difficult game states?

I don't think you've necessarily made any real mistakes in your calculations or anything like that, that all seems accurate. I can't really confidently answer your questions about "Does X usually ...
Dennis Soemers's user avatar
  • 10.2k
2 votes

Is which sense was AlphaGo "just given a rule book"?

it was "just given the rulebook", what does this mean? Literally a book written in English to read? The program was not given a natural language version of the rules to interpret. That ...
Neil Slater's user avatar
  • 31.5k
2 votes
Accepted

How does Alpha Go Zero MCTS work in parallel?

To understand how AlphaGo Zero performs parallel simulations think of each simulation as a separate agent that interacts with the search tree. Each agent starts from the root node and selects an ...
DSarkar's user avatar
  • 218
2 votes
Accepted

What do simulations mean in the context of the AlphaGoZero paper?

Your observation is accurate. The use of the term "simulation" in the AlphaGoZero paper can indeed be confusing. It does not refer to the simulation or rollout phase of the Monte Carlo Tree ...
Cesar Ruiz's user avatar
1 vote

Why is tree search/planning used in reinforcement learning?

Some sources say MCTS (or planning in general) increases the sample efficiency. If we're thinking purely about experiments run in simulations, then I'd estimate there may be cases where a combination ...
Dennis Soemers's user avatar
  • 10.2k
1 vote

What is the search depth of AlphaGo and AlphaGo Zero?

For easier visualization, I recommend this video: https://twitter.com/i/status/1257053365424578565 The more detailed article about GO algorithms: https://deepmind.com/blog/article/alphago-zero-...
Piotr Żak's user avatar
1 vote

Is it possible for AlphaGo Zero to use recurrent networks to achieve similar performance?

I can't think of any reason why using an RNN wouldn't work in theory. In practice RNNs are slightly harder to train (they can be unstable, and ever more practically you have to deal with multiple ...
Todd Sewell's user avatar
1 vote

Does AlphaGo play random moves in a real competition?

AlphaGo/AlphaZero has 3 main sources of randomness during competitive mode: Move temperature: The MCTS process outputs a probability distribution P over all candidate moves. The agent chooses a move ...
dshin's user avatar
  • 161
1 vote

Does AlphaGo play random moves in a real competition?

The core mechanics of AlphaZero during selfplay and real tournament games are the same: something similar to Monte Carlo Tree Search is done but guided by the current neural network instead of random ...
Todd Sewell's user avatar
1 vote
Accepted

How does the AlphaGo Zero policy decide what move to execute?

The formula in question uses a function N(state, action) that defines a visit count of a state-action pair (introduced on page 3). To describe how it is used, lets first describe the steps of AlphaGo ...
Jaden Travnik's user avatar
1 vote

How do the achievements met in the gaming field (ex. AlphaGo Zero) impact other fields of application?

Yes it's created something important. Until Alpha(Go) Zero all (or almost all) of Deep Learning approach to Reinforcement Learning was based on Time Difference loss function. The weakness of Time ...
mirror2image's user avatar
1 vote

What is the difference between DQN and AlphaGo Zero?

You can actually combine AlphaZero-like approach with DQN: A* + DQN
mirror2image's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible