I'm reading paper "Fictitious Self-Play in Extensive-Form Games", which introduces fictitious self-play(FPS).

In extensive-form games, let $\beta$ be the best response strategy, $\pi$ be the average strategy over a history of the best response strategies. We have the update rule for the average strategy as $\sigma=(1-\eta)\pi+\eta\beta$, where $\eta$ is the step size. In that paper, FPS uses two sources of data for reinforcement learning.

  1. data sampled by the average strategy profile, i.e., $(\sigma^i,\sigma^{-i})$
  2. data sampled by the best response strategy against opponent's average strategy profile, i.e., $(\beta^i,\sigma^{-i})$

My question is why FSP uses the data sampled by the average strategy profile $(\sigma^i,\sigma^{-i})$ for reinforcement learning? Why not use the data sampled by $(\beta^i,\sigma^{-i})$ only, which to my best knowledge is more suitable for training an RL algorithm due to the on-policy nature(even for an off-policy reinforcement learning method like Q-learning)?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.