Skip to main content
Share Your Experience: Take the 2024 Developer Survey

Questions tagged [language-model]

For questions related to the concept of a language model, which is a probability distribution over sequences of words (for example, of a natural language, such as English).

29 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
3 votes
0 answers
116 views

How do I balance context and history when creating prompts for LLM's?

A conversation through the OpenAI API looks something like this ...
Ian Purton's user avatar
3 votes
0 answers
2k views

What is input (and shape) to K/V/Q of self-attention of EACH Decoder block of Language-translation model Transformer's tokens during Inference?

Transformer model of the original Attention paper has a decoder unit that works differently during Inference than Tranining. I'm trying to understand the shapes used during decoder (both self-...
Joe Black's user avatar
  • 181
2 votes
1 answer
83 views

Is the problem of Language Modelling a Well-Posed Learning Problem?

Hadamard defines (Well-posed problem (Wikipedia)) a well-posed problem as one for which: a solution exists, the solution is unique, the solution depends continuously on the data (e.g. it is stable) ...
aren't eistert's user avatar
2 votes
0 answers
384 views

What's the most efficient way of performing batched training of Causal Language Models?

I have seen a number of ways to train (yes, train, not fine-tune) these models efficiently with batches. I will illustrate these techniques with the following example dataset and context window: ...
thesofakillers's user avatar
2 votes
0 answers
80 views

How can I generate a document from a single word using GPT or BERT?

I have a dataset of 100000 documents each labelled with a topic to it. I want to create a model such that, given a topic, the model can generate a document from it. I came across language models GPT,...
mayank agrawal's user avatar
1 vote
1 answer
31 views

publically available language models that can be used to train arbitrary language data?

I have sentence data in a language that is not widely in use and as such popular LLMs do not support the language. I want to train some language model such that given some question, it is able to ...
Neijal Kanderbalt's user avatar
1 vote
1 answer
107 views

Why do current language models no longer generate to long or short texts?

One of the biggest strengths of ChatGPT is that it generates fitting text with respect to the input query. It usually stays on topic, anwers the question completely and especially does not start ...
Ricu's user avatar
  • 11
1 vote
0 answers
43 views

Can I reduce computation by only predicting response tokens in a transformer and still get the same gradients?

I have been looking at the source code of the Stanford Alpaca model and I believe that during inference, the whole instruction + response data is fed into the model normally. Then the instruction part ...
Tianchen Zheng's user avatar
1 vote
0 answers
89 views

Coding a conversational AI which remembers previous context

I am trying to code a proper conversational AI which remembers previous context and answers accordingly (something like a micro ChatGPT). Additionally I want the AI to work on a custom knowledge base ...
JAYDEEP GHOSE's user avatar
1 vote
0 answers
796 views

How to best implement Accelerate for vary large models

This was pointed out to me: https://huggingface.co/blog/accelerate-large-models and seems it could be a great resource, but it is broken down into steps as a tutorial vs as out of the box solution ...
jmhead's user avatar
  • 111
1 vote
0 answers
87 views

How can i create a new language model for language other than english?

I have large set of corpus for all literature in 'Tamil' language, i am trying to create a document retrieval engine through simple natural language. Since the corpus is huge, its hard to do a ...
Thiyaga B's user avatar
  • 111
1 vote
0 answers
23 views

How can (pretrained) language models actively seek additional training data - possibly reference request?

I am reading the paper "Large Language Models Can Self-Improve" https://arxiv.org/abs/2210.11610 in which the authors consider that LLM can generate Chain-of-Thoughts sequences and even ...
TomR's user avatar
  • 853
1 vote
0 answers
27 views

Are custom tokens better than punctuation pseudo-tokens for LLMs?

I've seen two approaches for introducing custom tokens for transfer learning with large language models like Bert or GPT3. Some approaches introduce new tokens into the vocabulary and learn embeddings ...
A Tyshka's user avatar
  • 111
1 vote
0 answers
25 views

Which language model technique should be used with a low-resource language?

I am working with a language for which the amount of text available is relatively small (less than a billion words), what techniques exist? In particular, is it worth using a transformer or is it ...
user989070's user avatar
1 vote
0 answers
11 views

When we translate a text from one language to another, how does the frequency of various POS tags change?

When we translate a text from one language to another, how does the frequency of various POS tags change? So, let's say we have a text in English, with 10% nouns, 20% adjectives, 15% adverbs, 25% ...
Kushal Shah's user avatar
1 vote
0 answers
175 views

How is input defined for a biaxial lstm network for generating music?

I am reading Composing Music With Recurrent Neural Networks by Daniel D. Johnson. But I am really confused about the input passed to this network. If we pass notes of music along the time axis, then ...
Dracula's user avatar
  • 111
1 vote
0 answers
79 views

How to pad sequences during training for an encoder decoder model

I've got an encoder-decoder model for character level English language spelling correction, it is pretty basic stuff with a two LSTM encoder and another LSTM decoder. However, up until now, I have ...
hhaefliger's user avatar
0 votes
0 answers
34 views

What type of AI model should I use to generate practice questions?

I have a set of multiple choice English questions, and I wanted to use AI to generate more questions to quiz myself with. I know there are platforms online to accomplish this, but I wanted to ...
vp27's user avatar
  • 1
0 votes
0 answers
76 views

Why do LLM tokenizers use a special symbol for space such as Ġ in BPE or ▁ in SPM?

Popular tokenizers use a special symbol such as "Ġ" (BPE) or "▁" (SentencePiece) to represent space. What is the reasoning behind this? I did try searching for the answer. I got ...
Borislav Stanimirov's user avatar
0 votes
0 answers
81 views

Seeking an LLM with Browser Control Capabilities for Website Interaction Use Cases

I'm exploring the possibility of using a large language model (LLM) that can interact with web browsers, including taking screenshots and controlling mouse and keyboard actions. Does anyone know of ...
medvedev1088's user avatar
0 votes
0 answers
44 views

How can Knowledge Graphs be Integrated with Language Models for Semantic Search?

I am exploring the incorporate knowledge graphs (KGs) with language models. I understand that KGs can provide structured understanding of entities and their relationships which can be crucial for ...
Exploring's user avatar
  • 343
0 votes
0 answers
109 views

Seeking Advice on Local AI Model for Analyzing Personal Files

I'm embarking on a project to develop an application for personal use that can assist me in retrieving specific information from my files. I'd like to be able to ask questions like "What was my ...
NoNam4's user avatar
  • 101
0 votes
0 answers
39 views

Create samples out of documents for Causal Language Modelling

I want to create an input source for Causal Language model using Llama 2 model in hugging face. I have a set of documents which are scraped from a specific website and want to fine-tune on them. Each ...
Dimits's user avatar
  • 1
0 votes
0 answers
22 views

Small language model vocabulary correction

I plan to train a small language model (less than 4B parameters) that can run on x86 and handle vocabulary correction such as: ...
Jeff Brower's user avatar
0 votes
0 answers
334 views

Optimal Quantity of Training Data for Fine-Tuning an LLM: Is Bigger Always Better?

I am currently working on fine-tuning an LLM for a specific task, and I am trying to determine the optimal size for my training dataset. Intuitively, one might think that the more data, the better. ...
Peyman's user avatar
  • 574
0 votes
0 answers
140 views

A technique to show what tokens are relatively predicted by an LLM

I’m picturing a technique where you can see what an LLM is likely to respond with, which updates in real time. It’s a bit trippy, but it’s like GitHub Copilot, in that there is predicted text while ...
Julius Hamilton's user avatar
0 votes
1 answer
621 views

How big the context can be using HuggingFace models?

I'm new on AI, Neural Networks, ChatBots and all this ecosystem. I'm trying to use a classical example of pre-trained models, more specifically ...
Magno C's user avatar
  • 101
0 votes
0 answers
12 views

When running reinforcement learning on a language model, how do you prevent the model from just minimizing its logits?

Suppose you have a pre-trained autoregressive language model, and some cost function C mapping strings to numbers. Lower costs mean a given generated string is "better". Pass the model some ...
Jack M's user avatar
  • 262
0 votes
0 answers
15 views

What is the next-character perplexity of the PaLM model?

In the 2022 paper "PaLM: Scaling Language Modeling with Pathways", what is the bits-per-character perplexity of the resultant pre-trained model for next-word prediction?
ejang's user avatar
  • 101