4
$\begingroup$

I wish to train two domain-specific models:

  • Domain 1: Constitution and related Legal Documents
  • Domain 2: Technical and related documents.

For Domain 1, I've access to a text-corpus with texts from the constitution and no question-context-answer tuples. For Domain 2, I've access to Question-Answer pairs.

Is it possible to fine-tune a light-weight BERT model for Question-Answering using just the data mentioned above?

If yes, what are the resources to achieve this task?

Some examples, from the huggingface/models library would be mrm8488/bert-tiny-5-finetuned-squadv2, sshleifer/tiny-distilbert-base-cased-distilled-squad, /twmkn9/albert-base-v2-squad2.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

The answer is yes but 'lightweight' will require a 'lightweight' model.

Your application for 'domain one' is called open domain question answering (ODQA). Here is a demonstration of ODQA using BERT: https://www.pragnakalp.com/demos/BERT-NLP-QnA-Demo/

Your application for 'domain two' is a little different. It is about learning sequences from sequences. More specifically these are called sequence to sequence models. Here is an example using a pre-trained BERT model fine-tuned on the Stanford Question Answering (SQuAD) dataset.

Stanford Question Answering Dataset (SQuAD) is a reading comprehension dataset, consisting of questions posed by crowdworkers on a set of Wikipedia articles, where the answer to every question is a segment of text, or span, from the corresponding reading passage, or the question might be unanswerable.

In both applications, the resources required are going to rely on the performance you require. There are many sizes of BERT models. Generally, the larger the model, the larger the GPU memory requirements, and the higher the performance (i.e. accuracy, precision, recall, F1-score, etc.). For example, I can run BERT Base on a particular data set on a GTX 1080Ti and a RTX 2080Ti but not BERT Large.

This article, NVIDIA Quadro RTX 6000 BERT Large Fine-tune Benchmarks with SQuAD Dataset shows performance for BERT using TensorFlow on four NVIDIA Quadro RTX 6000 GPUs.

There is a 'mobile' version of BERT called MobileBERT for running on small devices like smartphones. Here is an article on using that with SQuAD: https://www.tensorflow.org/lite/models/bert_qa/overview

cdQA-suite is a good package. The following should help in fine-tuning on your own corpus:

Improve this answer
$\endgroup$
7
  • $\begingroup$ 1) For Domain 1, I have a list of articles from which I want the model to answer questions, so it is Closed Domain, or rather I want it that way. 2) For Domain 2, yes I'm up to date with BERT and the memory issues, what I want to know specifically, is whether just a text corpus can be used to fine-tune a model. $\endgroup$
    – Anirban Saha
    Commented Jul 6, 2020 at 13:22
  • $\begingroup$ Yes, you only need a text corpus. $\endgroup$
    – Brian O'Donnell
    Commented Jul 6, 2020 at 13:32
  • $\begingroup$ Can you point me to a resource, using which I can fine-tune a model only using a text-corpus and then use this model for Question-Answering. $\endgroup$
    – Anirban Saha
    Commented Jul 6, 2020 at 13:47
  • $\begingroup$ I added some information on the cdQA Suite which should help. $\endgroup$
    – Brian O'Donnell
    Commented Jul 6, 2020 at 14:08
  • $\begingroup$ I've also used cdQA extensively, the problem with the same is the cdQA-annotator tool depends on us manually highlighting answer passages, which would be difficult to achieve when we have close to 5000 question-answer pairs. $\endgroup$
    – Anirban Saha
    Commented Jul 6, 2020 at 14:14

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .