1
$\begingroup$

I was reading the NFSP player from D. Silver, and I'm somewhat confused by the algorithm:

enter image description here

In particular, given that we sample an action according to best response ($\sigma = \epsilon-\text{greedy}(Q)$), we also insert this transition in $\mathcal{M}_{RL}$, over which then we will estimate the gradient for the policy $\pi$... however, since this action has not been sampled from the policy $\pi$, it biases the gradient, which usually should be corrected by the importance sampling ratio

what am I missing?

$\endgroup$

1 Answer 1

3
$\begingroup$

over which then we will estimate the gradient for the policy π ...

It seems like you might be misunderstanding the type of algorithm. This (NFSP) doesn't use return of $\Pi$ as a loss. If Heinrich & Silver were using a policy gradient method, they would, you would encounter the issue of the gradient being off-policy.

This is actually strictly a Q Learning algorithm. The policy of the MDP is epsilon-greedy based on our Q estimator. The policy function $\Pi$ is an estimator for $\epsilon\text{-greedy}(Q)$, and is trained using SL according to negative-log-likelihood.

What you would need to be careful of when implementing this NFSP is when you clear the replay memories. If you update your Q-network parameters $\theta^{Q} \leftarrow \theta^{Q'}$ (an easy mistake) and continue sampling, your trajectory distribution will be different. However, the algorithm described in the paper is entirely on-policy.

$\endgroup$
3
  • $\begingroup$ oh I see, infact the SL loss of the actor was just a binary crossentropy (and not an actor critic loss), so $\pi$ is just trying to approximate the best response $\endgroup$
    – Alberto
    Commented Jul 19, 2023 at 13:17
  • $\begingroup$ For some reason, I always though FSP was and Actor Critic method where the best actor would go against old versions of itself, how strange $\endgroup$
    – Alberto
    Commented Jul 19, 2023 at 13:18
  • $\begingroup$ @AlbertoSinigaglia AlphaStar used both FSP and used policy gradient; NFSP being a Q learning method is probably owed to it being from 2016, when policy gradient wasn't as common. $\endgroup$ Commented Jul 19, 2023 at 13:27

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .