5
$\begingroup$

I have already implemented a relatively simple DQN on Pacman.

Now I would like to clearly understand the difference between a DQN and the techniques used by AlphaGo zero/AlphaZero and I couldn't find a place where the features of both approaches are compared.

Also sometimes, when reading through blogs, I believe different terms might in fact be the same mathematical tool which adds to the difficulty of clearly understanding the differences. For example, variations of DQN e.g. Double DQN also uses two networks like alpha zero.

Has someone a good reference regarding this question ? Be it a book or an online ressource.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
6
$\begingroup$

DQN and AlphaZero do not share much in terms of implementation.

However, they are based on the same Reinforcement Learning (RL) theoretical framework. If you understand terms like MDP, reward, return, value, policy, then these are interchangeable between DQN and AlphaZero. When it comes to implementation, and what each part of the system is doing, then this is less interchangeable. For instance two networks you have read about in AlphaZero are the policy network and value network. Whilst double DQN alternates between two value networks.

Probably the best resource that summarises both DQN and AlphaZero, and explains how they extend the basic RL framework in different ways is Sutton & Barto's Reinforcement Learning: An Introduction (second edition) - Chapter 16 sections 5 and 6 cover the designs of DQN Atari, AlphaGo and AlphaZero in some depth.

In brief:

DQN Atari

  • Is model-free
  • Uses an action value estimator for $Q(s,a)$ values, based on a Convolutional Neural Network (CNN)
  • Uses experience replay and temporarily frozen target network to stabilise learning process
  • Uses a variety of tricks to simplify and standardise the state description and reward structure so that the exact same design and hyperparameters work across multiple games, demonstrating that it is a general learner.

AlphaZero

  • Is model based (although some of the learning is technically model-free, based on samples of play)
  • Uses a policy network (estimating $\pi(a|s)$) and a state value network (estimating $V(s)$), based on CNNs. In practice for efficiency the NN for these share many layers and parameters, so how many "networks" there are depends how you want to count them.
    • The earlier AlphaGo version had 4 separate networks, 3 variations of policy network - used during play at different stages of planning - and one value network.
  • Is designed around self-play
  • Uses Monte Carlo Tree Search (MCTS) as part of estimating returns - MCTS is a planning algorithm critical to AlphaZero's success, and there is no equivalent component in DQN
Improve this answer
$\endgroup$
5
  • $\begingroup$ Note that in practice we usually just use a single network for the policy $\pi$ as well as the value estimates $V$, just with different heads for the different outputs (most of the weights are shared) $\endgroup$
    – Dennis Soemers ♦
    Commented Feb 28, 2019 at 17:11
  • $\begingroup$ @DennisSoemers: Yes. IIRC, there is a second simplified policy network as well in AlphaGo Zero, in order to run a "rollout policy" quickly. I'm skating over the detail a little. $\endgroup$
    – Neil Slater
    Commented Feb 28, 2019 at 21:08
  • $\begingroup$ I think that was only AlphaGo, not AlphaGo Zero anymore :) I'm quite sure the simplified policy in AlphaGo was not even really a network anymore, just a linear function + softmax. AlphaGo Zero doesn't really do rollouts anymore, so it doesn't need that either. $\endgroup$
    – Dennis Soemers ♦
    Commented Mar 1, 2019 at 9:08
  • 1
    $\begingroup$ @DennisSoemers: I just re-read the chapters in Sutton & Barto. From what I understand of it, there were 4 separate networks in AlphaGo (3 variations of Policy and one Value network), and just one combined network in AlphaGo Zero. $\endgroup$
    – Neil Slater
    Commented Mar 13, 2019 at 9:16
  • $\begingroup$ Ah yeah that's right, AlphaGo had the SL policy network, RL policy network, and Value network. Well, and the rollout policy, but I don't count that as a network (it's just linear + softmax). There was something about the SL policy network vs RL policy network mentioned in the paper, can't remember 100% for sure what it was. Think it was something like that the RL Policy network was better for generating experience to train the Value Network, but the SL policy network was still better at biasing the Selection phase (due to having higher entropy). Something like that... but not 100% sure $\endgroup$
    – Dennis Soemers ♦
    Commented Mar 13, 2019 at 9:56
1
$\begingroup$

You can actually combine AlphaZero-like approach with DQN: A* + DQN

Improve this answer
$\endgroup$
2
  • $\begingroup$ The A* algorithm in that link is the pathfinding / graph search algorithm, that's something completely different from AlphaZero (or even the AlphaStar program for StarCraft) $\endgroup$
    – Dennis Soemers ♦
    Commented Feb 28, 2019 at 17:10
  • 1
    $\begingroup$ It has a lot in common with AlphaZero. Both use tree search for generating training samples and deep network for action evaluation at the node. The main difference is AlephZero use Monte Carlo tree search and this algo use A* tree search. The common main idea is that tree branching allow reduce extrapolation error of RL methods. AlphaStar on the other hand has little in common with AlphaZero. It's just policy gradient algorithm $\endgroup$
    – mirror2image
    Commented Mar 1, 2019 at 7:40

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .