1
$\begingroup$

In Appendix B of MuZero, they say

In two-player zero-sum games the value functions are assumed to be bounded within the $[0, 1]$ interval.

I'm confused about the boundary: Shouldn't the value/utility function be in the range of [-1,1] for two-player zero-sum games?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

it can be either. If you consider the lack of reward as "penalty" then getting 0 reward is bad.

if you use a value estimator through a neural network, the range of rewards will dictate the squashing function you use for the output layer

Improve this answer
$\endgroup$
19
  • 1
    $\begingroup$ Hi, @nikos if we use [0,1] as the bound for the value functions, then how can the game be zero-sum? $\endgroup$
    – Maybe
    Commented May 6, 2020 at 1:07
  • $\begingroup$ The utility function is valued from the perspective of ONE of the 2 players, so it definitely shouldn't sum to zero! If you train it properly, it should give a larger percentage (probability) to the AI player winning. You should regard "zero sum" in the sense one wins and the other loses $\endgroup$
    – nikos
    Commented May 7, 2020 at 10:50
  • $\begingroup$ Hi @nikos. The game will return either $+1$ or $-1$ to the agent to indicate whether it wins or not. I get that the agent is unlikely to take actions that cause it to lose, but I cannot see any guarantee that restricts the value function to positive. For example, if the agent is in a state where there is no way to win if its opponent acts optimally. Shouldn't the value function of that state be negative in that case? $\endgroup$
    – Maybe
    Commented May 8, 2020 at 11:48
  • 1
    $\begingroup$ I haven't read the particular paper that you refer to, but what I've been trying to tell you all along is that the actual range of rewards doesn't matter. If you want [-1,1], that's fine, and [0,1] is also fine, as well as [-1000,1000]. You just have to be consistent $\endgroup$
    – nikos
    Commented May 12, 2020 at 11:04
  • 1
    $\begingroup$ I got that, but this range is not an arbitrary choice; it affects the value of pUCT, which is important to MCTS. $\endgroup$
    – Maybe
    Commented May 13, 2020 at 1:00

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .