The paper is here https://arxiv.org/pdf/1906.00190.pdf
and the relevant paragraph where they explain their method is below:
It's still not clear to me how this is meant to work exactly. In the pseudo-code, the formula for the new NeuRD policy update is given in the all actions case $\theta_{t+1} \leftarrow \theta_{t} + \eta \sum_{a} \nabla_{\theta} y(a) \ A_{\pi}(a) $, the components of which are just the gradient of each logit ($y(a)$) times the advantage of that action. The NeuRD update rule as I understand it is just deep exp3.
But then is the paragraph from the paper meaning that: for each minibatch and each $a, y(a)$, we compute the new model parameters $\theta_{t+1, a}$ which are given by using only the term in the original update corresponding to $a$. Then we see if the logits $y_{\theta_{t+1, a}}(a)$ of the minibatch forward pass are out of bounds $[-B, B]$ (either the mean or max logit). If the logit is out of bounds, we do not use that component of the update. Note this not prevent logits from being pushed out of bounds by and update from another action.
This is my simplest iterpretation of their description and it seems terribly expensive. I'd have to update and un-update the network $k$ times for each minibatch.
At least in the tabular case (the NeuRD update rule is just exp3), clipping is necessary to prevent a logit from diverging to $- \infty$ for an action that is dominated. This context also suggests we can't "cheat" by just considering if the whole update pushes any logit out of bounds, and scrapping the update if so. In the tabular case, this method would eventually zero all gradients once any logit gets pushed out.
Also I'd like to comment that "gradient clipping" is a poor choice of name. "logit clipping" seems from accurate.
