1
$\begingroup$

There is a nice post about the intuition why AlphaZero works.

One of the advantages of using a policy network in the games where a perfect simulator is available (such as chess) is to save computation time by not generating all subsequent moves and then evaluating them using the value network. Instead, we can only focus on the good moves given by the policy network.

However, besides the computation time savings of the policy network, are there any requirements why it needs to be used during training?

What if we would replace the computation of the policy network with this logic: generate all subsequent moves, evaluate them using value network, and create policy from these predictions. Would this still work?

I would appreciate any references where this topic is discussed.

$\endgroup$
1
$\begingroup$

I'm not aware of anyone running a setup of everything that AlphaZero does, minus the Policy Network, and reporting on how well it worked, so I don't think I can provide a definitive 100% certain answer. My intuition says that it would "work" in the sense that it could still produce a very strong agent, but I suspect it could be slower to train and/or not reach as high of a peak.


One important advantage of the Policy Network over the Value Network is specifically during the training phase, and in particular in the beginning of training; the Policy Network receives much more training data. When we run self-play games as in the AlphaZero paper, we get one target for the Policy Network for every distinct state we encounter (so many different targets per state), but only a single target for the Value Network for every full game (just the outcome of the game). There are papers that try to address this in different ways (some extract multiple value targets from the tree, some speed up the generation of self-play games by only running shorter searches in some states and not generating policy targets from the shorter searches, etc.), but ignoring those... in the standard AlphaZero setup, we may expect the Policy Network to train faster / more easily, especially in the beginning. The Value Network can "catch up" afterwards.

A second advantage of the Policy Network over the Value Network is that it may more effectively learn to distinguish good moves from bad moves in highly one-sided game states. Consider a game state $s$ where the player is (almost) sure to win, and can even afford to make a few mistakes. We'll probably get that all the successor states $s'$ get approximately the same value estimates $V(s') \approx 0.999$, with only a tiny amount of variation, and we may be unable to distinguish good moves from bad moves. The value network may no longer have any sort of noticeable preference for good moves over bad moves, until it has made so many blunders/mistakes that it actually becomes necessary to play well again. It would probably still win, but possibly in a less convincing manner (and slower) than it could. It's not even wrong for the value network to learn in this way, since we do want it (ideally) to learn the game-theoretic values, and this would be correct from that point of view. But probably still undesirable. In contrast, the Policy Network is not trained to predict game-theoretic values, or trained just to win. It's actively trained to pick good moves over bad moves. Due to the exploratory behaviour that is inherent in MCTS, MCTS generally ends up preferring more convincing / faster / safer wins, so the Policy Network does actually get an incentive to really play well even in such states where mistakes can be afforded.

A similar story to the above also applies to game states where the player is almost sure to lose; the Value Network gets no incentive to distinguish states from each other and might end up playing poorly in losing states, whereas a Policy Network still gets incentive to play well and "fight back", which may end up allowing it to still reach a win against suboptimal players.


What if we would replace the computation of the policy network with this logic: generate all subsequent moves, evaluate them using value network, and create policy from these predictions.

This particular implementation you suggest would have an additional disadvantage of computational inefficiency: to evaluate the Value Network for all possible successor states, you actually need to 1) generate all those successor states first and 2) run forwards passes of the Neural Network for every successor. Both of those things take time. In contrast, the Policy Network just runs once for the single parent state, and at once computes all the probabilities for all the actions. This is much more efficient.


Related to your suggestion, but quite a bit different because it does not use MCTS (but a different tree search algorithm) as the underlying search algorithm, a rather effective approach that only uses a Value Network (no Policy Network) is described in "Learning to Play Two-Player Perfect-Information Games without Knowledge" by Cohen-Solal, with additional empirical results described in "Minimax Strikes Back".

$\endgroup$
4
  • $\begingroup$ Thank you for a nice answer! It seems that Minimax Strikes Back is exactly what I was looking for. Just one note to "Policy Network receives much more training data". If I am not mistaken the number of states used for training is the same for policy and value network, is not in? Also the targets for value networks should be equally distributed in the training samples; half of them are -1 and the other half is +1 (if there is no draw). So it is not really clear to me why policy network should learn faster. Can you please elaborate on that or point me to the mentioned literature? Many thanks! $\endgroup$
    – Druudik
    Nov 12 '21 at 1:05
  • $\begingroup$ @Druudik Technically yeah, in your experience buffer you get equal amounts of samples for Policy and Value. But the policy targets are different for every state, and specifically "made" for that particular state. The value targets are the same for all states in the full game (ignoring negation for the opposite colours), and really not "made" for that particular state. They're probably fairly accurate for the last few states, but may be really inaccurate for early states. I recommend taking a look at Section 3.1 of arxiv.org/abs/1902.10565 $\endgroup$
    – Dennis Soemers
    Nov 12 '21 at 8:34
  • $\begingroup$ I think I am starting to get what you meant by that. Once again many thanks! $\endgroup$
    – Druudik
    Nov 12 '21 at 14:47
  • $\begingroup$ Just adding reference to one more article that I've just found which I think explains exactly what you were mentioning medium.com/oracledevs/…. $\endgroup$
    – Druudik
    Nov 12 '21 at 15:16
0
$\begingroup$

The solution approach for the linked work Mastering the game of Go with deep neural networks and tree search is not only valid for the game Go but can also be used for other games, e.g. chess. The policy network has the task to learn the rules of the game and is then improved only with the help of the value network by focusing on winning games. If I understand you correctly, you want to evaluate all the moves using the value network and pick the best one? For this you need rules - i.e. which moves are allowed to be made at all. That's what I think you need the policy network for.

$\endgroup$
1
  • $\begingroup$ Regarding the rules: that is what I've meant by "where perfect simulator is available". Would policy network be requirement also in this case? $\endgroup$
    – Druudik
    Nov 11 '21 at 15:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.