Skip to main content

Questions tagged [tokenization]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
1 answer
43 views

Why Tokenization Algorithm replace space with underscore "_"?

I have recently read an article for Byte-Pair Encoding (BPE), in which they have replaced space with an underscore(_), or we can say, adding underscore after each word/token. I did not understand the ...
Aamod Thakur's user avatar
0 votes
0 answers
10 views

Calculating prediction confidence from a sequence of token-level confidences

I am working with OCSR (optical chemical structure recognition) models, and they output a sequence of token-level confidences. I am looking for a method of summarising these token-level confidences ...
finlay morrison's user avatar
0 votes
0 answers
51 views

Is there research on the effect of typos in LLM prompts?

A simple typo can split a single token for a common word into several tokens, not only making the prompt longer, but also creating a combination of tokens that was rare in the training set. I wonder ...
allo's user avatar
  • 310