I was wondering if the BERT or T5 models can do the task of generating sentences in English. Most of the models I have mentioned are trained to translate from English to German or French. Is it possible that I can use the output of BERT as an input to my Decoder? My theory is that when I already have the trained Embeddings, I do not need to train the Encoder part. I can just add the outputs of sentences to the decoder to generate the sentences.

In place of finding the loss value from the translated version, Can I compute loss on the reply of a sentence?

Can someone point me toward a tutorial where I can use the BERT output for the decoder part? I have a data of conversation with me. I want to build a Chatbot from that data.

I have already implemented LSTM based Sequence2sequence model but it is not providing me satisfactory answer.

After some research, 2 such models are there as T5 and BART which are based on the same idea.

If possible, can someone tell me how can I use BART or T5 to make a conversational bot?


1 Answer 1


=> I don't have more ideas about BART and T5 Right now. but I had created a chatbot Based on GPT-2 Model-based on Microsoft DialoGPT. Which is fine-tuned on millions of parameters of Reddit. you can fine-tune on your own data using DialoGPT. I had not found a public decoding method. So, We tried to generate diverse responses using simple nucleus sampling at each time step. So, We used greedy nucleus sampling multiple times in parallel to generate multiple candidates responses. We generated 30 such responses for each turn and also used the last 3 turn conversation history as extra context for generator. So, We can generate responses related to the current Context of Conversation.

We used Code reference from here. you need to implement greedy nucleus sampling multiple on this code.

=>Now To Choose the best-matched response from response generated I created a Reranker.I used two sub-component.

=>Component one Counts Cosine similarity between Response and Query using sentence Encoder.

=>In Component Two we cross-entropy error for generating original query from the response.For that I used reverse-generator of DialogueGPT.

We Combined Score of Both Component as Score=norm(comp1Score)+norm(1-comp2Loss). Using this score you can find best response.

For Tutorial My Favourite Hugging Face they have BERT,GPT, GPT-2, XLNET,etc... and Second Favourite Facebook ParlAI, try Blender Model also.

  • $\begingroup$ But can we use outputs of BERT as the the K' and 'V values to an attention Decoder in the simplest terms possible to minimize the binary cross entropy? $\endgroup$
    – Deshwal
    Jun 12, 2020 at 7:14
  • $\begingroup$ @HirenNamera Can you please use this style only for code? You can and should use this or this style if you need to emphasize something. $\endgroup$
    – nbro
    Jun 12, 2020 at 11:03
  • $\begingroup$ I think it is possible to minimize binary cross entropy. Why not you try using huggingface's pretrainedencoderdecoder module. Using bert-base-uncased as encoder and gpt-2 large as a decoder. $\endgroup$ Jun 12, 2020 at 11:10

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .