I'm new on AI, Neural Networks, ChatBots and all this ecosystem. I'm trying to use a classical example of pre-trained models, more specifically timpal0l/mdeberta-v3-base-squad2.

As I could see in the examples, it is necessary to provide a small text (context) and a question, which the model will respond to by extracting information from the context and using training on the structure of the language. This is my working code so far:

from transformers import AutoTokenizer, AutoModelForQuestionAnswering, AutoConfig, DefaultDataCollator
import torch

tokenizer = AutoTokenizer.from_pretrained("timpal0l/mdeberta-v3-base-squad2")
model = AutoModelForQuestionAnswering.from_pretrained("timpal0l/mdeberta-v3-base-squad2", return_dict=False )

# From portuguese: My cat is called Helena. She is fat and has spots all over her body.
little_text = "minha gata se chama helena. Ela é gorda e tem manchas pelo corpo."

# From portuguese: who is helena?
question = "quem é helena?"

inputs = tokenizer.encode_plus(question, little_text, add_special_tokens=False, return_tensors="pt")

input_ids = inputs["input_ids"].tolist()[0]

text_tokens = tokenizer.convert_ids_to_tokens(input_ids)
answer_start_scores, answer_end_scores = model(**inputs)

answer_start = torch.argmax(
answer_end = torch.argmax(answer_end_scores) + 1

answer = tokenizer.convert_tokens_to_string(tokenizer.convert_ids_to_tokens(input_ids[answer_start:answer_end]))

print(f"Question: {question}")
print(f"Answer: {answer}\n")

Question: quem é helena?
Answer: minha gata # From portuguese: My cat 

Everything went well. The model is able to perfectly answer questions about the small text provided as context.

What I couldn't understand is: when I provide a much larger text to serve as context (from a file, for example) the time for the response increases considerably. So every time I go to ask a question about the given text I have to wait a long time to get the answer.

So my question is: Am I doing this correctly? My intention is to create a chatbot to help answer questions about a specific system (a kind of help bot), so I simply copied all the text from the online help and pasted it into a single file but I believe that this method only works for small texts because at every question I must tokenizer.encode_plus(question, little_text ...

How big the context can be?

  • $\begingroup$ When working with large texts as context, the time for the response can indeed increase significantly. This is because the model needs to process and understand the entire context before generating an answer. The larger the context, the more time it will take for the model to encode and analyze it. ▰ To avoid reprocessing the entire context for each question, please preprocess the context separately and encode it once, storing the representation for later. Then, for each question, you can encode only the question and pass it along with the preprocessed context to the model for inference. $\endgroup$ Jun 8, 2023 at 13:55
  • $\begingroup$ Can you please show where I must change my code to achieve this? Asked to GPT but the answer was a terrible mess. $\endgroup$
    – Magno C
    Jun 8, 2023 at 20:01
  • $\begingroup$ @MagnoC Don't rely on ChatGPT for this kind of questions. ChatGPT is only relatively good at generating new random text that is in a way related to some text that you pass to it. ChatGPT is can be bad at providing info about facts and especially specific facts (remember that it only produces the most likely tokens and it doesn't really reason like us). Having said that, models usually have a fixed context, including ChatGPT (which should have like 4096 context length). How long is your text? $\endgroup$
    – nbro
    Jun 9, 2023 at 8:57
  • $\begingroup$ Anyway, this question seems to be specific to a specific pre-trained model, so it seems to be off-topic. See our on-topic page: ai.stackexchange.com/help/on-topic. We focus on the theory. $\endgroup$
    – nbro
    Jun 9, 2023 at 8:59
  • $\begingroup$ Ok. Thanks anyway. But @RevolucionforMonica gives me a solution that is generic for this KIND of network and not for this specific model because any model can be used with this code ( I think ) $\endgroup$
    – Magno C
    Jun 10, 2023 at 15:21

1 Answer 1


The Attention computation is contrains by square of the contexte. Tokenize is mandatory and avoid to tokenize / detokenize too many time can help. Your RAM or VRam is alsoo limitation as memory size is... square of the context.

  • $\begingroup$ Thanks, but my question was edited by @nbro and lost the meaning. I do not care how big the context can be. My question was about how can I work with big contexts without need to wait a life for the answer. The key was given by "Revolucion for Monica" with the word "preprocess" $\endgroup$
    – Magno C
    Jan 23 at 11:50

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .