2
$\begingroup$

Alphago and AlphaGo zero use random play to generate data and use the data to train DNN. "Random play" means that there is a positive probability for AlphaGo to play some suboptimal moves based on the current DNN; this is for exploring and learning purposes (please correct me if my understanding is wrong).

In the real tournament, does AlphaGo still play the random moves? Is the random play feature only used in the training phase?

If AlphaGo does not play a random move in the real competition, then I think AlphaGo is not learning in that competition. Human players do similar "random play": they usually play some random moves or strange moves in minor contests, just to test out new strategies; in major tournaments, they will be more serious and play less unprepared moves.

So, a related and broader question is: does AlphaGo learn from the game it is playing with the human in real-time?

I think the second question is less important because AlphaGo's learning curve is extremely flat compared to humans: AlphaGo learns epsilon from one single game while human can learn a lot.

$\endgroup$
1
  • $\begingroup$ Please, ask only one question per post, even if they are related! $\endgroup$
    – nbro
    Jan 23 at 22:24

2 Answers 2

2
$\begingroup$

Question 1: I don't think they ran AlphaGo or AlphaGoZero in training mode during tournament matches because the computing power required for this is really large. I don't recall if this is described in the documentary but see this quote from the AlphaZero paper (page 4):

using 5,000 first-generation TPUs (15) to generate self-play games and 64 second-generation TPUs to train the neural networks.

Question 2: From my first answer, I assume the network is run in inference mode, i.e., taking the observation of the environment as input and outputting the action (next move) according to the strongest strategy of the network. Then the AlphaGo does not learn anything as it does not evaluate whether the taken action is better than the strategy because these are the same. The way that the network learns during training/exploration is by comparing the random action that is taken to the best action known by the network.


You are correct in your understanding of self-play and exploration. This is done in RL to explore actions that might not have been taken yet to explore actions that might be better than the best-known strategy.

But during inference, you would want to greedily take the best-known actions because an action taken at random would most likely be way worse.

$\endgroup$
6
  • 1
    $\begingroup$ I think you are right for 1 and 2, that the NN was not being trained. However, my understanding is that MCTS was still used to improve on the NN policies, so it is not as cut and dry as this answer implies. $\endgroup$ Jan 19 at 8:11
  • $\begingroup$ That is true and an important note which I did not consider. MCTS is used in AlphaGo and AlphaZero and described in the respective papers. Although its use is quite limited compared to traditional alpha-beta search engines, such as the Stockfish chess engine. $\endgroup$
    – Lars
    Jan 19 at 8:45
  • $\begingroup$ Many thanks for your contribution! So in the "inference mode", AlphaGo will only choose the "best move"? What if there are two equally best move with the same value? $\endgroup$
    – High GPA
    Jan 22 at 18:50
  • $\begingroup$ To my knowledge, I don't think there can be two moves with equal value unless your discount factor is 0, or close to it. With a discount factor of 0, the agent will not consider the value of the future moves which can be determined from the immediate action the agent is about to take. But hypothetically speaking, it would probably be a random selection between actions of equal value. $\endgroup$
    – Lars
    Jan 24 at 7:09
  • $\begingroup$ @Lars Note that the Go board is symmetric so there are some scenarios that two or more moves are absolutely symmetric. If those symmetric moves are not of same value, then the algorithm has a problem. So, in my humble opinion, my question still hold: when two moves have exactly the same value, which move will be chosen? $\endgroup$
    – High GPA
    Jan 26 at 8:08
1
$\begingroup$

The core mechanics of AlphaZero during selfplay and real tournament games are the same: something similar to Monte Carlo Tree Search is done but guided by the current neural network instead of random simulations. The network is only doing inference, it's not learning during a tree search. There's a great summary diagram here.

The differences between selfplay and tournaments are:

  • Selfplay tree searches games get some additional (Dirichlet) noise injected to encourage exploration of different moves and positions. This does not happen during tournament games since that would likely make play weaker.

  • During selfplay moves are selected randomly, but weighed according to how good the tree search thinks certain moves are. This means the entire training process spends most of its time looking at good moves, but still occasionally plays bad moves to keep some variety. During tournaments AlphaZero just plays what it thinks is the best move.

  • Selfplay games are kept in a large buffer with millions of positions, on which the next neural network is trained. This is the only point at which actual network training is happening.

  • Selfplay tree searches only look at ~800 nodes per position, which probably only takes a few milliseconds to calculate. Tournament games take a lot longer, 3h per game + 15s per move. This means the tree search visits hundreds of thousands, likely millions of positions for each move, making tournament play a lot stronger than selfplay.

These are the only differences between selfplay and tournament games. One thing to note is that after selfplay/training is done, AlphaZero is just an immutable neural network with some tree search code around it. There is no learning happening during a tournament, or any additional randomness injected into the process. Instead, to ensure a diversity of games, an Opening Book is used.

$\endgroup$
4
  • $\begingroup$ I think MCTS is inherently random, unless you have some statement about that being fixed in some way by Deep Mind for tournament play? $\endgroup$ Jan 24 at 13:16
  • $\begingroup$ @NeilSlater Pure MCTS is indeed random, since it uses random simulations as the evaluation function. AlphaZero instead uses one of the outputs of the neural network as the value estimate. This makes AlphaZero with a fixed node budget fully deterministic, except for maybe some implementation details (multithreading, GPU batch processing, ...). $\endgroup$ Jan 24 at 14:17
  • $\begingroup$ MCTS also makes random choices earlier than rollout (Selection and Expansion phases). I don't think AlphaZero's truncated rollout fully removes randomness. $\endgroup$ Jan 24 at 14:20
  • $\begingroup$ @NeilSlater There is some varierty in MCTS details, but the vanilla one also has deterministic selection and expansion (which is not actually a distinct step IMO). Regardless, from the AlphaGo paper, in the description of figure 2, it's pretty clear that selection is deterministically based only on previous neural network outputs. $\endgroup$ Jan 24 at 14:28

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .