7

The output of the policy network is as described in the original paper: A move in chess may be described in two parts: selecting the piece to move, and then selecting among the legal moves for that piece. We represent the policy π(a|s) by a 8 × 8 × 73 stack of planes encoding a probability distribution over 4,672 possible moves. Each of the 8×8 ...


6

DQN and AlphaZero do not share much in terms of implementation. However, they are based on the same Reinforcement Learning (RL) theoretical framework. If you understand terms like MDP, reward, return, value, policy, then these are interchangeable between DQN and AlphaZero. When it comes to implementation, and what each part of the system is doing, then ...


6

Why has this merge proven beneficial? If you think about the shared Value/Policy network as consisting of a shared component (the Residual Network layers) with a Value and Policy component on top rather than Separation of Concerns it makes more sense. The underlying premise is that the shared part of the network (the ResNet) provides a high-level ...


4

Assuming you mean a mathematically perfect player, similar to what we can achieve trivially in Tic Tac Toe, then the answer is "maybe". The underlying reinforcement learning algorithms that it uses do have some convergence guarantees, but there are some caveats: Theories of convergence that apply to value and policy functions learned by RL assume ...


3

To my understanding, this is basically a supervised learning problem, where from the self play we have games associated with their winners, and the network is being trained to map game states to likelihood of winning. Yes, although the data for this supervised learning problem was provided by self-play. As AlphaZero learned, the board evaluations of the ...


3

We cannot tell with certainty whether AlphaGo Zero would become perfect with enough training time. This is because none of the parts (Neural Network) that would benefit from infinite training time (= a nice approximation of "enough" training time) are guaranteed to ever converge to a perfect solution. The main limiting factor is that we do not know whether ...


3

If you learn a policy or a value function from experience (that is, interaction with an environment), that's RL. In the case of AlphaGo, the MCTS is used to acquire the experience. RL could in fact be considered supervised learning (SL) or, more specifically, self-supervised learning, where the experience corresponds to the labels in SL, especially nowadays ...


1

The input to the neural network is a $19 × 19 × 17$ image stack comprising $17$ binary feature planes. $8$ feature planes $X_t$ consist of binary values indicating the presence of the current player’s stones ($X^i_t = 1$ if intersection $i$ contains a stone of the player’s colour at time-step $t$; $0$ if the intersection is empty, contains an opponent ...


1

Yes it's created something important. Until Alpha(Go) Zero all (or almost all) of Deep Learning approach to Reinforcement Learning was based on Time Difference loss function. The weakness of Time Difference loss function was that it was essentially training on itself, that is data produced by the same method was used as part of regression target. That was ...


1

The paper that introduced AlphaGo, Mastering the game of Go with deep neural networks and tree search, motivates the use of MCTS Monte Carlo tree search (MCTS) uses Monte Carlo rollouts to estimate the value of each state in a search tree. As more simulations are executed, the search tree grows larger and the relevant values become more accurate. The ...


1

You can actually combine AlphaZero-like approach with DQN: A* + DQN


1

Yes AlphaGo Zero could become undeniably perfect. It has won 100:0 against AlphaGo Lee (which won 4:1 against 18-time world champion (human) Lee Sedol) and 89:11 against AlphaGo Master (which won 60 straight online games against human professional Go players from 29 December 2016 to 4 January 2017). From the official AlphaGo website: "AlphaGo's 4-1 ...


Only top voted, non community-wiki answers of a minimum length are eligible