How do I do further (domain-specific) pre-training with Google BERT in preparation for subsequent fine-tuning?

Another way to say this is: can you create a checkpoint file created from the final output of BERT?

The paper How to Fine-Tune BERT for Text Classification? talks about this additional fine-tuning.

  • $\begingroup$ 1) load model, 2) train model, 3) save-out model $\endgroup$ – mshlis Jul 24 '19 at 12:47
  • $\begingroup$ @mshlis thanks -- the final output of bert lists these files at the end of this message. Have you successfully called init_from_checkpoint() using this? The checkpoint output is subtly different and the framework I was using, bert-as-service, was not happy when I pointed it at the base model to treat it as a checkpoint. bert_config.json bert_model.ckpt.data-00000-of-00001 bert_model.ckpt.index bert_model.ckpt.meta vocab.txt model.ckpt-343.data-00000-of-00001 model.ckpt-343.index model.ckpt-343.meta $\endgroup$ – Julian H Jul 25 '19 at 12:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.