This question is connected to a question that I asked some time ago.

This is how I understood the training procedure takes place (please correct any conceptual mistakes here):

  1. Many complete games are generated entirely with Monte-Carlo Tree Search (MCTS) and a fixed policy/value/state network. Then we iterate many times the following steps:
  2. A random game is sampled and a random position at step $t$ in the game is picked
  3. The network plays $K$ steps, at each step producing a policy $\mathbf{p}_{t+k}$ and a value $v_{t+k}$.
  4. At the same time the MCTS produces a policy $\boldsymbol{\pi}_{t+k}$.

The way MuZero computes the loss function is: $$ L_t = \sum_{k=1}^K \underbrace{(u - v_{t+k})^2}_\mathrm{value\ loss} + \underbrace{\boldsymbol{\pi}_{t+k} \log \mathbf{p}_{t+k}}_\mathrm{policy\ loss} + \underbrace{||\theta||^2}_\mathrm{L2\ regularization} $$ $u$ is the observed final reward of the episode, which is known because the episode in memory goes until the end.

I have two major doubts:

  1. why should one use $u$, if the moves that we end up picking over those $K$ steps can deviate from the moves in the original game that we sampled?
  2. if one player wins and one player loses, shouldn't we add the cross entropy of the policy for the winner and subtract the cross entropy of the policy of the loser? Is the answer that in any case the MCTS policy $\boldsymbol{\pi}_{t+k}$ is "stronger" than the bare network policy $\mathbf{p}_{t+k}$ and therefore we can still learn from it regardless if that player won or lost?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.