0
$\begingroup$

"[SEP] tokens are useful to differentiate the questions from answers through type_ids" Yes, but how is this helping model to understand that "I should look paragraph and generate answers from here"? We don't have if-else inside the model that will say: "if type_id==1, generate questions from here"

The same question appears for this example too:

[CLS] previous question [Q] current question [SEP] text [EOS]

How model says: "I should look at the previous question, understand the meaning from here too with the current question, and answer the question." We need if-else in here too like:

If there is a previous question: get the meaning from it and use it with the current question

One more example based on this paper:https://arxiv.org/pdf/2005.01107v1.pdf In this paper, we have a dataset like this: https://huggingface.co/datasets/iarfmoose/question_generator

In the t5 transformer, if we don't have those and tokens, the model will not learn anything. And I have no idea how specifying and we are helping the model to generate questions based on that answer in the context.

I hope I am clear with my question.

$\endgroup$

1 Answer 1

2
$\begingroup$

I think you are thinking too much like traditional programming here. A neural network doesn't need if-else. Providing the special tokens hint to the model about the structure of the question, and with enough data and training, the language model learns to utilize that hint.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .