Skip to main content

New answers tagged

0 votes

How big the context can be using HuggingFace models?

Context and Model Overview: Model Capabilities: The timpal0l/mdeberta-v3-base-squad2 model, seem to be based on DeBERTa V3, utilizes relative position embeddings ...
Kulin Patel's user avatar
0 votes

Why do current language models no longer generate to long or short texts?

TLDR: if you're using a model without instruction-tuning, you'll likely need to do some additional work during inference to make sure that your outputs are coherent regardless of how old your model is....
Alexander Wan's user avatar
0 votes

Classifier-Free-Guidance with Transformers

Cross-attention reduces to self-attention
Sahil Goyal's user avatar
0 votes

How are perplexities over multiple instance aggregated?

The choice of arithmetic mean for averaging perplexities is mainly for simplicity and consistency. It provides a straightforward measure of overall model performance across multiple sequences. Each $...
Kulin Patel's user avatar
0 votes

is bfloat training compatible with float16 inference?

Let us know the difference between bfloat16 and float16. https://blogs.nvidia.com/blog/tensorfloat-32-precision-format/ As per the above blog, bfloat16 has more exponent bits than float16, which means ...
Aamod Thakur's user avatar
1 vote

Any popular diffusion model for language modeling?

The problem with the current diffusion models, is that they are great density models, but they require to know a priori the size of the sample (ie the size of the image), which definitely is not the ...
Alberto's user avatar
  • 2,283
0 votes

probability intepreter of attention mechanism in Seq2Seq

Pretty much anything can be interpreted as a distribution or some kind of energy model, that doesn't mean that everything is the same First of all, the softmax output of a Seq2Seq is just a ...
Alberto's user avatar
  • 2,283
0 votes

understanding the distribution shift problem in direct preference optimization (DPO)

I had a problem concerning this and I think I can shed some light onto your question. We had a dataset of preferences generated for a really bad model, so naturally $y_w$ and $y_l$ were bad responses. ...
Guilherme Correa's user avatar
0 votes
Accepted

Why Tokenization Algorithm replace space with underscore "_"?

As you have spotted, there is no particular reason to choose a specific character. However, the author of the article is operating under some constraints that I can guess have guided the choices: The ...
Neil Slater's user avatar
  • 32.7k
0 votes

Train my own LLM on a smaller corpus of text?

If you have an OpenAI Pro account you can define your own GPT that can do certain operations. If your stuff is publicly accessible, this is one way to get started quickly: Upload the documentation ...
knb's user avatar
  • 165

Top 50 recent answers are included