I am thinking about a project and have a few questions before I accept it. Would be grateful I anyone experienced of you could give me some advice.

In the project, I have been given a data set with (rather small) 30.000 text documents, which are labeled with 0 and 1. I want to train and evaluate (with respect to accuracy) a BERT and XLNet model.

Can you give me some rough estimates for the following questions?:

  1. How much computing power do I need for this task, i.e. can I simply use my private laptop for this or do I need a special CPU/GPU for it?
  2. So far, I just worked with classical machine learning models (e.g. random forests, SVMs, etc.). I am not experienced deep learning architectures yet. How difficult would it be to implement a BERT oder XLNet model with my own data set, having no experience with BERT oder XLNet yet? I.e. how much code would it be that I have to develop by myself? And would I need a deep understanding for it or would be sufficient to follow an online tutorial and basically copy the code from there? Many thanks.
  1. You’ll want a reasonable GPU (probably 8GB+), but otherwise no special hardware needed.* You may need to tune down sequence length and batch size to fit your GPU; RAM will be the limiting factor. Don’t try it on a CPU. It will “work” but you’re gonna have a bad time.
  2. Try the Huggingface Transformers library as your implementation. It’s well documented and straightforward and includes both models.

*assuming an Nvida GPU or something compatible with CUDA. Things are rather hairier on Apple hardware. But you can always grab a cloud VM for a few hours


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.