7
$\begingroup$

I'm looking to make an NLP model that can achieve a dual purpose. One purpose is that it can hold interesting conversations (conversational AI), and another being that it can do intent classification and even accomplish the classified task.

To accomplish this, would I need to use multimodal machine learning, where you combine the signal from two models into one? Or can it be done with a single model?

In my internet searches, I found BERT, developed by Google engineers (although apparently not a Google product), which is an NLP model trained in an unsupervised fashion on 3.3 billion words or more and seems very capable.

How can I leverage BERT to make my own conversational AI that can also carry out tasks? Is it as simple as copying the weights from BERT to your own model?

Any guidance is appreciated.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

Think of BERT (or similar models) as as good starting place for understanding context.

A couple options to make BERT contextualize dialogue:

  • Concatenate all messages with a seperator embedding and finetune a language model like BERT
    • This has shown good results in this paper, but understand it has weaknesses like struggling to determine order or author comprehension or conversation disentanglement. Additionally as threads grow longer this becomes more and more expensive
  • Use A BERT like model as an RNN and pass along author/memory information as the hidden state
    • Weaknesses here is that you are putting a lot of strain on the hidden state of the model, and ideally you want to train each comment block separately so the gradient wouldnt need to unroll or traverse the network in backprop (I actually implemented a variant of this for my work, and had good results)

The 2 modes above are just baseline examples that you can alter for your own needs, and im sure there is also others if you brainstorm about it.

Note if you want generation capabilities, you would want to use GPT2 or XLNet instead of BERT for unidirectional embeddings. Hope this helped.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .