Can we use a pre trained Encoder (BERT, XLM ) with a Decoder (GPT, Transformer-XL) to build a Chatbot instead of Language Translation?

Question

I was wondering if the BERT or T5 models can do the task of generating sentences in English. Most of the models I have mentioned are trained to translate from English to German or French. Is it possible that I can use the output of BERT as an input to my Decoder? My theory is that when I already have the trained Embeddings, I do not need to train the Encoder part. I can just add the outputs of sentences to the decoder to generate the sentences.

In place of finding the loss value from the translated version, Can I compute loss on the reply of a sentence?

Can someone point me toward a tutorial where I can use the BERT output for the decoder part? I have a data of conversation with me. I want to build a Chatbot from that data.

I have already implemented LSTM based Sequence2sequence model but it is not providing me satisfactory answer.

After some research, 2 such models are there as T5 and BART which are based on the same idea.

If possible, can someone tell me how can I use BART or T5 to make a conversational bot?

Answer 1

=> I don't have more ideas about BART and T5 Right now. but I had created a chatbot Based on GPT-2 Model-based on Microsoft DialoGPT. Which is fine-tuned on millions of parameters of Reddit. you can fine-tune on your own data using DialoGPT. I had not found a public decoding method. So, We tried to generate diverse responses using simple nucleus sampling at each time step. So, We used greedy nucleus sampling multiple times in parallel to generate multiple candidates responses. We generated 30 such responses for each turn and also used the last 3 turn conversation history as extra context for generator. So, We can generate responses related to the current Context of Conversation.

We used Code reference from here. you need to implement greedy nucleus sampling multiple on this code.

=>Now To Choose the best-matched response from response generated I created a Reranker.I used two sub-component.

=>Component one Counts Cosine similarity between Response and Query using sentence Encoder.

=>In Component Two we cross-entropy error for generating original query from the response.For that I used reverse-generator of DialogueGPT.

We Combined Score of Both Component as Score=norm(comp1Score)+norm(1-comp2Loss). Using this score you can find best response.

For Tutorial My Favourite Hugging Face they have BERT,GPT, GPT-2, XLNET,etc... and Second Favourite Facebook ParlAI, try Blender Model also.