What are examples of simple gradient based NLP models?

Question

I am looking for a simple example of gradient-based methods for NLP. More specifically I am looking for post-hoc local explanations gradient-based methods, that is to say, which explain a single prediction by performing additional operations (after the model has emitted a prediction). Here is an example of a gradient-based method for "Finance"? (I'm not very sure of what domain it is as it is not NLP, nor Computer Vision, nor time series ...):

Let us consider a minimal example, where a linear regression is used to estimate the future capital asset $y_{c}$, based on two investments $x_{1}$ and $x_{2}$. Let assume the assumptions above are met and the model parameters are estimated as follows: $$\mathbb{E}\left[y_{c} \mid x_{1}, x_{2}\right]=1.05 x_{1}+1.50 x_{2},$$ We can derive immediately a global interpretation of this model. Every dollar invested in fund $x_{1}$ will produce capital of $1.05 \\\$$, while every dollar invested in $x_{2}$ will produce a capital of $1.50 \\\$$, independently of the values $x_{1}$ and $x_{2}$ might assume in a concrete scenario. Notice that this explanation is purely based on the learned coefficient $w_{1}=1.05$ and $w_{2}=1.50$. These, sometimes called partial regression coefficients, are themselves candidate attribution values to explain the influence of the independent variables of the target variable: $$R_{1}(x)=1.05 \quad R_{2}(x)=1.50$$ Notice also that the coefficients are the partial derivatives of the target variable with respect to the independent variable, therefore this attribution is nothing but the model gradient.

I thought about transformating it to an NLP example like:

Let us consider a minimal example, where a linear regression is used to estimate the toxicity of a text $y_{c}$, based on the occurence of words $x_{1}$ and $x_{2}$. Let's assume the assumptions above are met and the model parameters are estimated as follows:

$$\mathbb{E}\left[y_{c} \mid x_{1}, x_{2}\right]=1.05 x_{1}+1.50 x_{2},$$

We can derive immediately a global interpretation of this model. Every occurrence of word $x_{1}$ will produce a toxicity of $1.05$, while every occurrence of word in $x_{2}$ will produce a capital of $1.50$, independently of the values $x_{1}$ and $x_{2}$ might assume in a concrete scenario. Notice that this explanation is purely based on the learned coefficient $w_{1}=1.05$ and $w_{2}=1.50$. These, sometimes called partial regression coefficients, are themselves candidate attribution values to explain the influence of the independent variables of the target variable

That look pretty dumb, isn't it? Do you have any better example

Update : does the ablation model use gradient?

So it seems my model isn't so dumb, but Ant gave the idea of using the ablation model. I didn't know about this technique but it seems straightforward: you iterate over your text and get rid of a given word in part of the text and see how much the text gains or loses toxicity. It looked very gradient-friendly.

However, I tried to create an ablation algorithm that looks at the k next words and tries to get the toxicity of the word by doing the difference between the text predicted toxicity $\hat y$ and the same one when the text is being ablated $\hat y_{ablated}$. I inspired myself from Warren Freeborough's example for time series in his paper page 5.

I get an en error, or sensibility, matrix $E_{avg}$. But I don't see where any gradient has been used. At most I could say that the model can use gradient. For instance, if it is an RNN, it uses a gradient method, isn't it? But I don't see where ablation uses gradient.

Answer 1

Why does it look dumb? If you think the model makes sense, then the explanation seems reasonable to me.

I would not call this gradient-based explanation, however, since the features in this case are discrete, not continuous (you cannot take a derivative wrt to $x_1$ and $x_2$, if those are discrete features)

You can however look at ablations (i.e. change in scores if the word is not present), which would be the analogous of gradients for discrete features. The interpretation is: if that word is present, the score increases by this much. Why has this text been classified as toxic? Because it contains the following toxic words, each sorted by their "toxicity".

Of course that would be a very simple model; words do not occur in isolation and looking at their mere presence completely disregard the context in which they appear. However, that's a problem of the model, not of the explanation of the model.

The more sophisticated models these days use as base architecture the transformer, which is able to differentiate words based on the context in which they occur. There are some explanatory models for transformer-based architectures, but they will not simply assign a numerical score to a word (as that cannot accurately reflect the predictions of the model, since it ignores context).