"[SEP] tokens are useful to differentiate the questions from answers through type_ids" Yes, but how is this helping model to understand that "I should look paragraph and generate answers from here"? We don't have if-else inside the model that will say: "if type_id==1, generate questions from here"

The same question appears for this example too:

[CLS] previous question [Q] current question [SEP] text [EOS]

How model says: "I should look at the previous question, understand the meaning from here too with the current question, and answer the question." We need if-else in here too like:

If there is a previous question: get the meaning from it and use it with the current question

One more example based on this paper:https://arxiv.org/pdf/2005.01107v1.pdf In this paper, we have a dataset like this: https://huggingface.co/datasets/iarfmoose/question_generator

In the t5 transformer, if we don't have those and tokens, the model will not learn anything. And I have no idea how specifying and we are helping the model to generate questions based on that answer in the context.

I hope I am clear with my question.


1 Answer 1


I think you are thinking too much like traditional programming here. A neural network doesn't need if-else. Providing the special tokens hint to the model about the structure of the question, and with enough data and training, the language model learns to utilize that hint.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .