3
$\begingroup$

I'm coding my own version of MuZero. However, I don't understand how it supposed to learn to play well for both players in a two-player game.

Take Go for example. If I use a single MCTS to generate an entire game (to be used in the training stage), couldn't MuZero learn to play badly for black in order to become good at predicting a win for white? What is forcing it to play well at every turn?

Improve this question
$\endgroup$
1
  • $\begingroup$ Is is that the policies and values generated by the MCTS and the model are added to the loss for each "player" independently? (although in the paper it doesn't seem that it is like this) $\endgroup$
    – Ziofil
    Oct 31, 2020 at 18:51

1 Answer 1

Reset to default
1
$\begingroup$

Both players are represented by the exact same network with the exact same weights(similar to AplhaGO, AlphaGoZero and AlphaZero). So, they will both behave identical. Because you only have a single network, MuZero can not learn two different policies, but only one.

You can also think of this in the following way: MuZero actually learn to play only with white(or black, but just one of them) without knowing to play with the other color (at least multiple implementations of the previous algorithms like AlphaGo Zero and AlphaZero that are similar to MuZero are doing exactly this). So, in order to trick it to also be able to play with the other color, when the network need to play with black, you just flip the colors on the table so that black becomes white(and white becomes black) and the network knows what to do. After choosing the move, you flip the whole thing back and that is usually how it is done. So, from the perspective of your network, it will always play white, but because you do the flipping of the colors you can actually put them to play against each other without them knowing that.

Even without using this trick of flipping the color of the table, by doing the MCTS simulations, you will have for each state the statistics for the actions, and usually as you do more simulations, this statistics show you which actions are the best in each state. And when you train, you try to imitate this. So, your network will learn in each state which actions are the best, and this is the reason why it learns to take the best possible action in each state.

Improve this answer
$\endgroup$
6
  • $\begingroup$ Okay, but the agent is given information about whose turn it is, so the policy could still end up being turn-dependent. And in some sense it should be: some games like chess are asymmetric, so it doesn't make much sense to learn to play only from one point of view with the color flipping trick. I don't understand where the "best" actions are defined as best. In the policy loss they use the cross-entropy between the MCTS policy and the network policy, so maybe it comes from how the predicted values are backed up in the MCTS, but I'm not sure. $\endgroup$
    – Ziofil
    Oct 31, 2020 at 21:43
  • $\begingroup$ You are doing max over the actions. So, the network will always choose the best action. Moreover, past papers related to MuZero train the prediction on outcomes of matches, so the agents learn what will actually happen, and they can not make up the result, as they learn from experience. And the experience teaches them that some actions are good and some are bad, and given the way you select the actions using max, you will always choose the best action, given the amount of experience and knowledge the network has. So, the predicted outcome is actually learned and they can not cheat $\endgroup$
    – Otniel-Bogdan Mercea
    Oct 31, 2020 at 22:29
  • $\begingroup$ if one of the networks would learn to play badly, that means that it will learn to choose wrong actions, so that the max action would actually be the worst possible action. But that would mean that the loss between the true outcomes and the predicted outcomes will be big(as you need the outcome in order to choose which actions are the best in MCTS), so the optimizer would try to minimize that, and in the end you will reach the situation where the optimal solution is to predict a result which is as close as possible to reality. $\endgroup$
    – Otniel-Bogdan Mercea
    Oct 31, 2020 at 22:35
  • $\begingroup$ Well, yes I agree that in the MCTS you are doing argmax over the policy, but this alone is not a guarantee to get the best action: it just selects the action with highest "probability". In order for it to be the actual best action, we need the policy to be the probability of best action, and hence the question: how is the policy trained to be such? $\endgroup$
    – Ziofil
    Oct 31, 2020 at 23:27
  • $\begingroup$ Because you predict the true value(as said above), in the MCTS after many simulations, the action statistics depend on these predicted values. So if the predicted values are correct(and as said above they are because this is the only way of minimizing the loss), then the statistics of the actions will also reflect the true outcome of taking an action, so the highest probability will reflect the action that is providing the best outcome, because this is how MCTS works. You actually use the predicted values in computing the probability of each action in MCTS. $\endgroup$
    – Otniel-Bogdan Mercea
    Nov 1, 2020 at 11:32

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .