I am computing the attribution scores for a simple LSTM model using Integrated Gradients. This method defines the contribution of a feature to a model prediction by integrating over the gradients along a path between the input and a fixed baseline:

$$IG_i(x) = (x_i - x'_i) \cdot\int_{\alpha=0}^1 \frac{\partial F(x'+\alpha(x-x'))}{\partial x_i}d\alpha$$

A common way of measuring the quality of the generated attributions is via the completeness axiom, which states that:

$$\sum_i IG_i(x) = F(x) - F(x')$$

The key to computing the IG scores is the approximation of the path integral, which can be approximated via Riemann sum, or a similar interpolation method. In section 5 of the IG paper, it is stated that, in practice, between 50 to 300 interpolation steps are sufficient to obtain IG scores that converge to satisfy the completeness axiom.


I am now testing the IG attributions on a simple LSTM model (1-layer, 16 hidden units). For shorter inputs (<20 tokens), convergence is reached in a reasonable number of steps, and the approximation of the integral is stable. However, when the length of the input increases, I find that the integral approximation diverges when the number of interpolation steps is increased! This can be seen in the following plot (N.B. the y axis is logarithmic):

Delta scores for IG


My question is: why does the number of input tokens to the LSTM have an impact on the convergence of integrated gradients?

It is stated in footnote 1 of the IG paper that the completeness axiom depends on whether the model satisfies Lebesgue's integrability condition. It would surprise me, however, that increasing the number of input tokens would dissatisfy this constraint: would it be possible that the model has become too nonlinear for numerical integration to still work? If so, are there alternative numerical integration methods that could be used here, instead of Riemann approximations or the Gauss-Legendre quadrature?

  • 1
    $\begingroup$ Hello. Welcome to Artificial Intelligence SE. This seems to be an interesting question, even for people not familiar with IG. But can you clarify what you mean by "LSTM steps"? Do they refer to "interpolation steps"? If that's the case, what exactly do you mean by "interpolation steps" in this context? $\endgroup$
    – nbro
    Nov 4 '21 at 12:46
  • 1
    $\begingroup$ By LSTM step I mean a single forward step of an LSTM cell: so processing a single token and up to the next. Concretely: an input string consisting of 10 tokens would amount to processing 10 LSTM steps in order to obtain the final hidden state that is used to create a model prediction. $\endgroup$
    – jumelet
    Nov 4 '21 at 14:47
  • 1
    $\begingroup$ By interpolation steps I mean the steps in the sense of how a Riemann sum splits up the calculation of a path integral into smaller steps. The higher this number of steps is the more accurate the approximation of the integral will become. $\endgroup$
    – jumelet
    Nov 4 '21 at 14:55
  • $\begingroup$ I have updated the question a little bit to make the use of "steps" less confusing. Thanks also for your edits to the question, looks nice now :) $\endgroup$
    – jumelet
    Nov 4 '21 at 15:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.