The paper is here https://arxiv.org/pdf/1906.00190.pdf

and the relevant paragraph where they explain their method is below:

enter image description here

It's still not clear to me how this is meant to work exactly. In the pseudo-code, the formula for the new NeuRD policy update is given in the all actions case $\theta_{t+1} \leftarrow \theta_{t} + \eta \sum_{a} \nabla_{\theta} y(a) \ A_{\pi}(a) $, the components of which are just the gradient of each logit ($y(a)$) times the advantage of that action. The NeuRD update rule as I understand it is just deep exp3.

But then is the paragraph from the paper meaning that: for each minibatch and each $a, y(a)$, we compute the new model parameters $\theta_{t+1, a}$ which are given by using only the term in the original update corresponding to $a$. Then we see if the logits $y_{\theta_{t+1, a}}(a)$ of the minibatch forward pass are out of bounds $[-B, B]$ (either the mean or max logit). If the logit is out of bounds, we do not use that component of the update. Note this not prevent logits from being pushed out of bounds by and update from another action.

This is my simplest iterpretation of their description and it seems terribly expensive. I'd have to update and un-update the network $k$ times for each minibatch.

At least in the tabular case (the NeuRD update rule is just exp3), clipping is necessary to prevent a logit from diverging to $- \infty$ for an action that is dominated. This context also suggests we can't "cheat" by just considering if the whole update pushes any logit out of bounds, and scrapping the update if so. In the tabular case, this method would eventually zero all gradients once any logit gets pushed out.

Also I'd like to comment that "gradient clipping" is a poor choice of name. "logit clipping" seems from accurate.


1 Answer 1


An implementation of the 'gradient clipping' can be found in DeepMind's own OpenSpiel. https://github.com/deepmind/open_spiel/blob/master/open_spiel/python/algorithms/neurd.py

  • $\begingroup$ Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center. $\endgroup$
    – Community Bot
    Sep 8, 2022 at 19:07

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .