0
$\begingroup$

Are there any techniques to combine a feature set (other than the text itself) with pretrained language models.

Let's say I have a random NLP task that tries to predict a binary class label based on e.g. Twitter data. One could easily utilize a pretrained language model such as BERT/GPT-3 etc. to fine-tune it on the text of the tweets. However the tweets come with a lot of useful metadata such as likes/retweets etc. or if I want to add additional syntactic features such as POS-Tags, dependency relation or any other generated feature. Is it possible to use additional features I extracted for the finetuning step of the pretrained language model? Or is the only way of doing so to use an ensemble classifier and basically write a classifier for each of the extracted features and combine all of their predictions with the finetuned LMs predictions?

$\endgroup$

2 Answers 2

1
$\begingroup$

Although barely related, this brings to my mind a concept I first learned in conditional GANs - there, if you wanted an output of some specific class, you would take your random noise input $z$ (that is given to your generative model $G$) and concatenate it with some representation of the class instance. On the MNIST digits dataset, for example, you can provide $[z,(1,0,...,0)]$ where $(1,0,...,0)$ represents a one-hot encoding of the digit $0$. So perhaps you could use this concatenation concept for your task:

  1. extract features $\{a_i\}_{i=1}^N$ from the text itself - this can be done using a transformer, as suggested, or some RNN architecture like LSTM or GRU.
  2. extract features $\{b_i\}_{i=1}^N$ from the metadata - there are some approaches that deal with tabular data, but I would start with a basic MLP
  3. concatenate the two feature vectors $c_i=[a_i,b_i]$
  4. yield a final prediction using yet another model that takes $\{c_i\}_{i=1}^N$ as input

Edit: I've looked this up online, and found some supporting evidence - check out this blog post, for example.

$\endgroup$
1
$\begingroup$

If you're eventually building a classifier with added features, concatenating the LM output embeddings with those additional features should work, I believe.

It seems similar to incorporating non-sequential data, like location, in RNN models, where the final set of features for classification are last hidden layer output + non-sequential features.

incorporating non-sequential data

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .