Skip to main content
Share Your Experience: Take the 2024 Developer Survey

Questions tagged [language-model]

For questions related to the concept of a language model, which is a probability distribution over sequences of words (for example, of a natural language, such as English).

Filter by
Sorted by
Tagged with
6 votes
2 answers
578 views

How does GPT-based language model like ChatGPT determine the n-th letter of a word?

I understand that GPT models process input text by converting words into tokens and then embedding vectors and do not process them letter by letter. Given this approach, I am curious to know how a ...
1 vote
1 answer
31 views

publically available language models that can be used to train arbitrary language data?

I have sentence data in a language that is not widely in use and as such popular LLMs do not support the language. I want to train some language model such that given some question, it is able to ...
2 votes
1 answer
83 views

Is the problem of Language Modelling a Well-Posed Learning Problem?

Hadamard defines (Well-posed problem (Wikipedia)) a well-posed problem as one for which: a solution exists, the solution is unique, the solution depends continuously on the data (e.g. it is stable) ...
0 votes
0 answers
34 views

What type of AI model should I use to generate practice questions?

I have a set of multiple choice English questions, and I wanted to use AI to generate more questions to quiz myself with. I know there are platforms online to accomplish this, but I wanted to ...
0 votes
0 answers
81 views

Why do LLM tokenizers use a special symbol for space such as Ġ in BPE or ▁ in SPM?

Popular tokenizers use a special symbol such as "Ġ" (BPE) or "▁" (SentencePiece) to represent space. What is the reasoning behind this? I did try searching for the answer. I got ...
1 vote
2 answers
134 views

What technique is used for training Large Language Models like GPT?

I'm learning about GenAI, such as GPT (Generative Pretrained Transformer), and I'm particularly interested in understanding the training techniques used for these models. Deep learning, generally, can ...
0 votes
1 answer
107 views

What language model can convert normal text to JSON data

I have tried training T5-small, T5-base and T5-Large on around 15K rows of data where input data was something like but I did not get desired results ...
0 votes
1 answer
623 views

How big the context can be using HuggingFace models?

I'm new on AI, Neural Networks, ChatBots and all this ecosystem. I'm trying to use a classical example of pre-trained models, more specifically ...
1 vote
1 answer
108 views

Why do current language models no longer generate to long or short texts?

One of the biggest strengths of ChatGPT is that it generates fitting text with respect to the input query. It usually stays on topic, anwers the question completely and especially does not start ...
0 votes
0 answers
81 views

Seeking an LLM with Browser Control Capabilities for Website Interaction Use Cases

I'm exploring the possibility of using a large language model (LLM) that can interact with web browsers, including taking screenshots and controlling mouse and keyboard actions. Does anyone know of ...
2 votes
1 answer
180 views

Is beam search the actual obstacle that prevents GPT-style models from doing sophisticated math reasoning?

This is a rather soft question. Some people believe that GPT-style models can eventually solve very complex math problems if the models are large enough, but I'm skeptical about this. Suppose the GPT ...
2 votes
1 answer
64 views

NLP "small" model to improve "big" model

When training the model for NLP is it important to get rid of data which has "bad semantic" for learning process? My plan is to create a "small model" which can decide whether data ...
0 votes
1 answer
94 views

Setting number of rows returned by vector stores

When using vector stores like pinecone or Faiss from langchain, is it possible to set the number of records returned based on similarity search? For example, consider the following code, is there a ...
0 votes
0 answers
44 views

How can Knowledge Graphs be Integrated with Language Models for Semantic Search?

I am exploring the incorporate knowledge graphs (KGs) with language models. I understand that KGs can provide structured understanding of entities and their relationships which can be crucial for ...
1 vote
1 answer
80 views

What is the lowest possible loss for a language model?

Example: Suppose a character-level language model (three input letters to predict the next one), trained on a dataset which contains three instances of the sequence ...
1 vote
2 answers
53 views

Data preparation for NLP model

I have data from our ticketing system. Currently using OpenNLP to create different models. For simplicity I have a 10k ticket's text as category final queue of the ticket. My questions: Is it ...
3 votes
0 answers
116 views

How do I balance context and history when creating prompts for LLM's?

A conversation through the OpenAI API looks something like this ...
0 votes
0 answers
109 views

Seeking Advice on Local AI Model for Analyzing Personal Files

I'm embarking on a project to develop an application for personal use that can assist me in retrieving specific information from my files. I'd like to be able to ask questions like "What was my ...
0 votes
1 answer
174 views

Confusion About Triangle Mask in Transformer Decoder

I have some confusion about the implementation of the triangle mask in the transformer decoder. I understand the reasoning for the mask, it prevents the network from 'cheating' by looking ahead at the ...
3 votes
1 answer
106 views

How can I improve this toy Graph Neural Network Generative Language model [closed]

Background I'm an undergraduate student with research interests in a field of physics that has significant overlap with graph theory, and a functioning knowledge of how simple neural nets work and how ...
1 vote
1 answer
215 views

How to train a language model to write poem?

I am using some of the LLM applications, and all of them are better at provide formal, steady conversations, rather than writing texts with styles. So I wonder whether it is possible to train a LLM ...
0 votes
0 answers
39 views

Create samples out of documents for Causal Language Modelling

I want to create an input source for Causal Language model using Llama 2 model in hugging face. I have a set of documents which are scraped from a specific website and want to fine-tune on them. Each ...
1 vote
1 answer
158 views

Is there a limitation to the amount of data that a genAI model could be trained upon?

My friend says that genAI would become more human like, and perhaps even smarter than humans if it were simply trained on more and more data. I say that this would overtrain the models, and we would ...
0 votes
0 answers
22 views

Small language model vocabulary correction

I plan to train a small language model (less than 4B parameters) that can run on x86 and handle vocabulary correction such as: ...
0 votes
0 answers
334 views

Optimal Quantity of Training Data for Fine-Tuning an LLM: Is Bigger Always Better?

I am currently working on fine-tuning an LLM for a specific task, and I am trying to determine the optimal size for my training dataset. Intuitively, one might think that the more data, the better. ...
50 votes
3 answers
23k views

Was ChatGPT trained on Stack Overflow data?

Has ChatGPT used highly rated and upvoted questions/answers from Stack Overflow in its training data? For me it makes complete sense to take answers that have upwards of 100 upvotes and include them ...
0 votes
0 answers
141 views

A technique to show what tokens are relatively predicted by an LLM

I’m picturing a technique where you can see what an LLM is likely to respond with, which updates in real time. It’s a bit trippy, but it’s like GitHub Copilot, in that there is predicted text while ...
5 votes
2 answers
3k views

How is the next token predicted in transformers?

In the transformer (or GPT/decoder only), at the end of the decoder blocks but before the final linear layer you have X vectors (for the X tokens at the input of the decoder). We then want to compute ...
0 votes
0 answers
12 views

When running reinforcement learning on a language model, how do you prevent the model from just minimizing its logits?

Suppose you have a pre-trained autoregressive language model, and some cost function C mapping strings to numbers. Lower costs mean a given generated string is "better". Pass the model some ...
24 votes
4 answers
12k views

How does ChatGPT know math?

ChatGPT is a language model. As far as I know and If I'm not wrong, it gets text as tokens and word embeddings. So, how can it do math? For example, I asked: ME: Which one is bigger 5 or 9. ChatGPT: ...
36 votes
1 answer
41k views

What is the "temperature" in the GPT models?

What does the temperature parameter mean when talking about the GPT models? I know that a higher temperature value means more randomness, but I want to know how randomness is introduced. Does ...
1 vote
1 answer
196 views

If we prompt a large language model on a task, will its ability for other tasks be affected? How to recover?

For example, I guess that for some retrieval augmented LLMs, their generated contents may lack some creativity? Recent work has explored the inability of retrieval augmented methods to enhance the ...
6 votes
1 answer
779 views

How can a language model keep track of the provenance of the main knowledge/sources used to generate a given output?

One of the main criticisms against the use of ChatGPT on Stack Exchange is that it doesn't attribute the main knowledge/sources used to generate a given output. How can a language model keep track of ...
4 votes
1 answer
148 views

What puts the "chat" in a system like ChatGPT?

So I read Wolfram's What Is ChatGPT Doing … and Why Does It Work? but it left one really big question in my mind. His summary [if it could be called that!] really emphasizes that the core model is ...
4 votes
2 answers
486 views

What makes reproducing a model like GPT3/GPT3.5/ChatGPT difficult?

Is it difficult for other companies to train a model similar to ChatGPT, and what makes it difficult? What is challenging about reproducing the results obtained by OpenAI with ChatGPT/GPT3.5? Would it ...
0 votes
2 answers
865 views

Is the GPT-4 for text the same model that can input and output images?

Currently, the published GPT-4 can input and output text. A version of GPT-4 that can input and output text and images exists, according to the technical report, but is not yet publicly available. I ...
1 vote
1 answer
413 views

What is the difference between T5 and T0 models?

What is the difference between T5 and T0 models? I had read that T0 is T5 + LM. But as I know T5 uses encoder-decoder model like BART but BART can be used as LM so that's mean that T5 has a LM ...
3 votes
0 answers
2k views

What is input (and shape) to K/V/Q of self-attention of EACH Decoder block of Language-translation model Transformer's tokens during Inference?

Transformer model of the original Attention paper has a decoder unit that works differently during Inference than Tranining. I'm trying to understand the shapes used during decoder (both self-...
1 vote
0 answers
43 views

Can I reduce computation by only predicting response tokens in a transformer and still get the same gradients?

I have been looking at the source code of the Stanford Alpaca model and I believe that during inference, the whole instruction + response data is fed into the model normally. Then the instruction part ...
2 votes
0 answers
385 views

What's the most efficient way of performing batched training of Causal Language Models?

I have seen a number of ways to train (yes, train, not fine-tune) these models efficiently with batches. I will illustrate these techniques with the following example dataset and context window: ...
7 votes
2 answers
3k views

Why can't language models, like GPT-3, continuously learn once trained?

GPT-3 has a prompt limit of about ~2048 "tokens", which corresponds to about 4 characters in text. If my understanding is correct, a deep neural network is not learning after it is trained ...
0 votes
1 answer
5k views

How do temperature and repetition penalty interfere?

I'm trying to demystify my understanding of various decoding parameters. Building on our understanding of temperature, how does the repetition penalty interfere with temperature? For example, does ...
1 vote
0 answers
89 views

Coding a conversational AI which remembers previous context

I am trying to code a proper conversational AI which remembers previous context and answers accordingly (something like a micro ChatGPT). Additionally I want the AI to work on a custom knowledge base ...
16 votes
2 answers
7k views

Why does ChatGPT not give the answer text all at once?

When ChatGPT is generating an answer to my question, it generates it word by word. So I actually have to wait until I get the final answer. Is this just for show? Or is it really real-time generating ...
1 vote
0 answers
797 views

How to best implement Accelerate for vary large models

This was pointed out to me: https://huggingface.co/blog/accelerate-large-models and seems it could be a great resource, but it is broken down into steps as a tutorial vs as out of the box solution ...
1 vote
1 answer
160 views

What background should I have before starting to fine tune a Large Language Model?

I want to know what things I should be learning before trying to fine-tune or for that matter working with a large language model. In my case, I am trying to fine-tune bloom (https://huggingface.co/...
9 votes
1 answer
1k views

What causes ChatGPT to generate responses that refer to itself as a bot or LM?

ChatGPT occasionally generates responses to prompts that refer to itself as a "bot" or "language model." For instance, when given a certain input (the first paragraph of this ...
6 votes
1 answer
2k views

How was ChatGPT trained?

I know that large language models like GPT-3 are trained simply to continue pieces of text that have been scraped from the web. But how was ChatGPT trained, which, while also having a good ...
2 votes
1 answer
153 views

Would a transformer trained on highly specific material be as usable as a commercial product like ChatGPT?

Soft question here. I was recently learning a bit about how it is feasible to train a transformer on a personal computer like an M1 Mac. I have been told that the model could have 1-3 million ...
1 vote
1 answer
1k views

Papers on Prompt Engineering

I am into AI in general and NLP in particular. Besides, I have a background in philosophy, and the new LLMs like GPT-3 seem to have exciting capabilities. I want to study prompt engineering (for ...