0
$\begingroup$

I am unaware to use the derived checkpoints from pre-trained BERT model for the task of semantic text similarity.

!python create_pretraining_data.py \
          --input_file=/input_path/input_file.txt \
          --output_file=/tf_path/tf_examples.tfrecord \
          --vocab_file=/vocab_path/uncased_L-12_H-768_A-12/vocab.txt \
          --do_lower_case=True \
          --max_seq_length=128 \
          --max_predictions_per_seq=20 \
          --masked_lm_prob=0.15 \
          --random_seed=12345 \
          --dupe_factor=5

!python run_pretraining.py \
      --input_file=/tf_path/tf_examples.tfrecord \
      --output_dir=pretraining_output \
      --do_train=True \
      --do_eval=True \
      --bert_config_file=/bert_path/uncased_L-12_H-768_A-12/bert_config.json \
      --init_checkpoint=/bert_path/uncased_L-12_H-768_A-12/bert_model.ckpt\
      --train_batch_size=32 \
      --max_seq_length=128 \
      --max_predictions_per_seq=20 \
      --num_train_steps=20 \
      --num_warmup_steps=10 \
      --learning_rate=2e-5

I have run a pre-trained BERT model with some domain of corpora from scratch. I have got the checkpoints and graph.pbtxt file from the code above. But I am unaware on how to use those files for evaluating semantic text similarity test file.

$\endgroup$
1
$\begingroup$

Have a look at https://medium.com/the-artificial-impostor/news-topic-similarity-measure-using-pretrained-bert-model-1dbfe6a66f1d

You can have the two sentences as first and second use the next sentence score as a similarity measure. You can further fine-tune your model on some semantic similarity tasks like Sent-Eval or your own dataset if you have one

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.