How to use pretrained checkpoints of BERT model on semantic text similarity task?

Question

I am unaware to use the derived checkpoints from pre-trained BERT model for the task of semantic text similarity.

!python create_pretraining_data.py \
          --input_file=/input_path/input_file.txt \
          --output_file=/tf_path/tf_examples.tfrecord \
          --vocab_file=/vocab_path/uncased_L-12_H-768_A-12/vocab.txt \
          --do_lower_case=True \
          --max_seq_length=128 \
          --max_predictions_per_seq=20 \
          --masked_lm_prob=0.15 \
          --random_seed=12345 \
          --dupe_factor=5

!python run_pretraining.py \
      --input_file=/tf_path/tf_examples.tfrecord \
      --output_dir=pretraining_output \
      --do_train=True \
      --do_eval=True \
      --bert_config_file=/bert_path/uncased_L-12_H-768_A-12/bert_config.json \
      --init_checkpoint=/bert_path/uncased_L-12_H-768_A-12/bert_model.ckpt\
      --train_batch_size=32 \
      --max_seq_length=128 \
      --max_predictions_per_seq=20 \
      --num_train_steps=20 \
      --num_warmup_steps=10 \
      --learning_rate=2e-5

I have run a pre-trained BERT model with some domain of corpora from scratch. I have got the checkpoints and graph.pbtxt file from the code above. But I am unaware on how to use those files for evaluating semantic text similarity test file.

Answer 1

Have a look at https://medium.com/the-artificial-impostor/news-topic-similarity-measure-using-pretrained-bert-model-1dbfe6a66f1d

You can have the two sentences as first and second use the next sentence score as a similarity measure. You can further fine-tune your model on some semantic similarity tasks like Sent-Eval or your own dataset if you have one