1
$\begingroup$

In Appendix B of MuZero, they say

In two-player zero-sum games the value functions are assumed to be bounded within the $[0, 1]$ interval.

I'm confused about the boundary: Shouldn't the value/utility function be in the range of [-1,1] for two-player zero-sum games?

$\endgroup$
1
$\begingroup$

it can be either. If you consider the lack of reward as "penalty" then getting 0 reward is bad.

if you use a value estimator through a neural network, the range of rewards will dictate the squashing function you use for the output layer

| improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ Hi, @nikos if we use [0,1] as the bound for the value functions, then how can the game be zero-sum? $\endgroup$ – Maybe May 6 at 1:07
  • $\begingroup$ The utility function is valued from the perspective of ONE of the 2 players, so it definitely shouldn't sum to zero! If you train it properly, it should give a larger percentage (probability) to the AI player winning. You should regard "zero sum" in the sense one wins and the other loses $\endgroup$ – nikos May 7 at 10:50
  • $\begingroup$ Hi @nikos. The game will return either $+1$ or $-1$ to the agent to indicate whether it wins or not. I get that the agent is unlikely to take actions that cause it to lose, but I cannot see any guarantee that restricts the value function to positive. For example, if the agent is in a state where there is no way to win if its opponent acts optimally. Shouldn't the value function of that state be negative in that case? $\endgroup$ – Maybe May 8 at 11:48
  • 1
    $\begingroup$ I haven't read the particular paper that you refer to, but what I've been trying to tell you all along is that the actual range of rewards doesn't matter. If you want [-1,1], that's fine, and [0,1] is also fine, as well as [-1000,1000]. You just have to be consistent $\endgroup$ – nikos May 12 at 11:04
  • 1
    $\begingroup$ I got that, but this range is not an arbitrary choice; it affects the value of pUCT, which is important to MCTS. $\endgroup$ – Maybe May 13 at 1:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.