14

The output of the policy network is as described in the original paper: A move in chess may be described in two parts: selecting the piece to move, and then selecting among the legal moves for that piece. We represent the policy π(a|s) by a 8 × 8 × 73 stack of planes encoding a probability distribution over 4,672 possible moves. Each of the 8×8 ...


6

DQN and AlphaZero do not share much in terms of implementation. However, they are based on the same Reinforcement Learning (RL) theoretical framework. If you understand terms like MDP, reward, return, value, policy, then these are interchangeable between DQN and AlphaZero. When it comes to implementation, and what each part of the system is doing, then this ...


6

Why has this merge proven beneficial? If you think about the shared Value/Policy network as consisting of a shared component (the Residual Network layers) with a Value and Policy component on top rather than Separation of Concerns it makes more sense. The underlying premise is that the shared part of the network (the ResNet) provides a high-level ...


4

Did AlphaGo and AlphaGo [Zero] play 100 repetitions of the same sequence of boards, or were there 100 different games? There were 100 different games. You can view some example games between AlphaGo [Lee] and AlphaGo Zero here. They are clearly all different. This statement in the question shows a misunderstanding: My understanding of AlphaGo and AlphaGo [...


4

Assuming you mean a mathematically perfect player, similar to what we can achieve trivially in Tic Tac Toe, then the answer is "maybe". The underlying reinforcement learning algorithms that it uses do have some convergence guarantees, but there are some caveats: Theories of convergence that apply to value and policy functions learned by RL assume ...


3

To my understanding, this is basically a supervised learning problem, where from the self play we have games associated with their winners, and the network is being trained to map game states to likelihood of winning. Yes, although the data for this supervised learning problem was provided by self-play. As AlphaZero learned, the board evaluations of the ...


3

We cannot tell with certainty whether AlphaGo Zero would become perfect with enough training time. This is because none of the parts (Neural Network) that would benefit from infinite training time (= a nice approximation of "enough" training time) are guaranteed to ever converge to a perfect solution. The main limiting factor is that we do not know whether ...


3

If you learn a policy or a value function from experience (that is, interaction with an environment), that's RL. In the case of AlphaGo, the MCTS is used to acquire the experience. RL could in fact be considered supervised learning (SL) or, more specifically, self-supervised learning, where the experience corresponds to the labels in SL, especially nowadays ...


2

The input to the neural network is a $19 × 19 × 17$ image stack comprising $17$ binary feature planes. $8$ feature planes $X_t$ consist of binary values indicating the presence of the current player’s stones ($X^i_t = 1$ if intersection $i$ contains a stone of the player’s colour at time-step $t$; $0$ if the intersection is empty, contains an opponent ...


2

The answer is surprisingly hidden in the original AlphaGo paper: At the end of search AlphaGo selects the action with maximum visit count; this is less sensitive to outliers than maximizing action value. Unfortunately, there did not appear to be further details in the paper or in the related reference. The root child node (corresponding to an action) ...


1

For easier visualization, I recommend this video: https://twitter.com/i/status/1257053365424578565 The more detailed article about GO algorithms: https://deepmind.com/blog/article/alphago-zero-starting-scratch. With its breadth of $250$ possible moves each turn (go is played on a $19$ by $19$ board, compared to the much smaller $8$ by $8$ chess field) and a ...


1

The most important word for answering your question from that quote from the paper is probably the word "usually": These search probabilities usually select much stronger moves than the raw move probabilities $p$ of the neural network. It's not always going to be true, but more often than not / most of the time / "on average", we ...


1

The formula in question uses a function N(state, action) that defines a visit count of a state-action pair (introduced on page 3). To describe how it is used, lets first describe the steps of AlphaGo Zero as a whole. There are 4 "phases" to the Monte-Carlo tree search in AlphaGo Zero as depicted in Figure 2. The first 3 expand and update the tree and ...


1

How should I interpret the weights file of the Leela Zero neural network? ... However, I am not sure how to interpret the network weight file. The image shown in the question has a set of network parameters listed as a series of floating point values in ASCII or utf-8 on the right. The equivalent hexadecimal values for those characters are shown on the ...


1

Yes it's created something important. Until Alpha(Go) Zero all (or almost all) of Deep Learning approach to Reinforcement Learning was based on Time Difference loss function. The weakness of Time Difference loss function was that it was essentially training on itself, that is data produced by the same method was used as part of regression target. That was ...


1

The paper that introduced AlphaGo, Mastering the game of Go with deep neural networks and tree search, motivates the use of MCTS Monte Carlo tree search (MCTS) uses Monte Carlo rollouts to estimate the value of each state in a search tree. As more simulations are executed, the search tree grows larger and the relevant values become more accurate. The ...


1

You can actually combine AlphaZero-like approach with DQN: A* + DQN


1

Yes AlphaGo Zero could become undeniably perfect. It has won 100:0 against AlphaGo Lee (which won 4:1 against 18-time world champion (human) Lee Sedol) and 89:11 against AlphaGo Master (which won 60 straight online games against human professional Go players from 29 December 2016 to 4 January 2017). From the official AlphaGo website: "AlphaGo's 4-1 ...


Only top voted, non community-wiki answers of a minimum length are eligible