Questions tagged [language-model]

For questions related to the concept of a language model, which is a probability distribution over sequences of words (for example, of a natural language, such as English).

Filter by
Sorted by
Tagged with
0 votes
0 answers
32 views

What type of AI model should I use to generate practice questions?

I have a set of multiple choice English questions, and I wanted to use AI to generate more questions to quiz myself with. I know there are platforms online to accomplish this, but I wanted to ...
vp27's user avatar
  • 1
0 votes
0 answers
38 views

Why do LLM tokenizers use a special symbol for space such as Ġ in BPE or ▁ in SPM?

Popular tokenizers use a special symbol such as "Ġ" (BPE) or "▁" (SentencePiece) to represent space. What is the reasoning behind this? I did try searching for the answer. I got ...
Borislav Stanimirov's user avatar
1 vote
2 answers
93 views

What technique is used for training Large Language Models like GPT?

I'm learning about GenAI, such as GPT (Generative Pretrained Transformer), and I'm particularly interested in understanding the training techniques used for these models. Deep learning, generally, can ...
Exploring's user avatar
  • 343
0 votes
1 answer
65 views

What language model can convert normal text to JSON data

I have tried training T5-small, T5-base and T5-Large on around 15K rows of data where input data was something like but I did not get desired results ...
Arjun Singh's user avatar
0 votes
0 answers
64 views

Seeking an LLM with Browser Control Capabilities for Website Interaction Use Cases

I'm exploring the possibility of using a large language model (LLM) that can interact with web browsers, including taking screenshots and controlling mouse and keyboard actions. Does anyone know of ...
medvedev1088's user avatar
1 vote
1 answer
29 views

publically available language models that can be used to train arbitrary language data?

I have sentence data in a language that is not widely in use and as such popular LLMs do not support the language. I want to train some language model such that given some question, it is able to ...
Neijal Kanderbalt's user avatar
2 votes
1 answer
163 views

Is beam search the actual obstacle that prevents GPT-style models from doing sophisticated math reasoning?

This is a rather soft question. Some people believe that GPT-style models can eventually solve very complex math problems if the models are large enough, but I'm skeptical about this. Suppose the GPT ...
Soha's user avatar
  • 21
2 votes
1 answer
61 views

NLP "small" model to improve "big" model

When training the model for NLP is it important to get rid of data which has "bad semantic" for learning process? My plan is to create a "small model" which can decide whether data ...
Milkmaid's user avatar
  • 135
0 votes
0 answers
44 views

How can Knowledge Graphs be Integrated with Language Models for Semantic Search?

I am exploring the incorporate knowledge graphs (KGs) with language models. I understand that KGs can provide structured understanding of entities and their relationships which can be crucial for ...
Exploring's user avatar
  • 343
1 vote
1 answer
77 views

What is the lowest possible loss for a language model?

Example: Suppose a character-level language model (three input letters to predict the next one), trained on a dataset which contains three instances of the sequence ...
ViniciusArruda's user avatar
0 votes
1 answer
73 views

Setting number of rows returned by vector stores

When using vector stores like pinecone or Faiss from langchain, is it possible to set the number of records returned based on similarity search? For example, consider the following code, is there a ...
Karl 17302's user avatar
2 votes
0 answers
108 views

How do I balance context and history when creating prompts for LLM's?

A conversation through the OpenAI API looks something like this ...
Ian Purton's user avatar
1 vote
2 answers
52 views

Data preparation for NLP model

I have data from our ticketing system. Currently using OpenNLP to create different models. For simplicity I have a 10k ticket's text as category final queue of the ticket. My questions: Is it ...
Milkmaid's user avatar
  • 135
0 votes
0 answers
89 views

Seeking Advice on Local AI Model for Analyzing Personal Files

I'm embarking on a project to develop an application for personal use that can assist me in retrieving specific information from my files. I'd like to be able to ask questions like "What was my ...
NoNam4's user avatar
  • 101
1 vote
1 answer
96 views

Why do current language models no longer generate to long or short texts?

One of the biggest strengths of ChatGPT is that it generates fitting text with respect to the input query. It usually stays on topic, anwers the question completely and especially does not start ...
Ricu's user avatar
  • 11
1 vote
1 answer
174 views

How to train a language model to write poem?

I am using some of the LLM applications, and all of them are better at provide formal, steady conversations, rather than writing texts with styles. So I wonder whether it is possible to train a LLM ...
Cabbage's user avatar
  • 11
3 votes
1 answer
97 views

How can I improve this toy Graph Neural Network Generative Language model [closed]

Background I'm an undergraduate student with research interests in a field of physics that has significant overlap with graph theory, and a functioning knowledge of how simple neural nets work and how ...
MomentumEigenstate's user avatar
0 votes
0 answers
39 views

Create samples out of documents for Causal Language Modelling

I want to create an input source for Causal Language model using Llama 2 model in hugging face. I have a set of documents which are scraped from a specific website and want to fine-tune on them. Each ...
Dimits's user avatar
  • 1
0 votes
0 answers
21 views

Small language model vocabulary correction

I plan to train a small language model (less than 4B parameters) that can run on x86 and handle vocabulary correction such as: ...
Jeff Brower's user avatar
0 votes
0 answers
333 views

Optimal Quantity of Training Data for Fine-Tuning an LLM: Is Bigger Always Better?

I am currently working on fine-tuning an LLM for a specific task, and I am trying to determine the optimal size for my training dataset. Intuitively, one might think that the more data, the better. ...
Peyman's user avatar
  • 564
0 votes
1 answer
154 views

Confusion About Triangle Mask in Transformer Decoder

I have some confusion about the implementation of the triangle mask in the transformer decoder. I understand the reasoning for the mask, it prevents the network from 'cheating' by looking ahead at the ...
new2java's user avatar
1 vote
1 answer
143 views

Is there a limitation to the amount of data that a genAI model could be trained upon?

My friend says that genAI would become more human like, and perhaps even smarter than humans if it were simply trained on more and more data. I say that this would overtrain the models, and we would ...
tryst with freedom's user avatar
0 votes
0 answers
133 views

A technique to show what tokens are relatively predicted by an LLM

I’m picturing a technique where you can see what an LLM is likely to respond with, which updates in real time. It’s a bit trippy, but it’s like GitHub Copilot, in that there is predicted text while ...
Julius Hamilton's user avatar
0 votes
1 answer
563 views

How big the context can be using HuggingFace models?

I'm new on AI, Neural Networks, ChatBots and all this ecosystem. I'm trying to use a classical example of pre-trained models, more specifically ...
Magno C's user avatar
  • 101
0 votes
0 answers
12 views

When running reinforcement learning on a language model, how do you prevent the model from just minimizing its logits?

Suppose you have a pre-trained autoregressive language model, and some cost function C mapping strings to numbers. Lower costs mean a given generated string is "better". Pass the model some ...
Jack M's user avatar
  • 262
2 votes
1 answer
83 views

Is the problem of Language Modelling a Well-Posed Learning Problem?

Hadamard defines (Well-posed problem (Wikipedia)) a well-posed problem as one for which: a solution exists, the solution is unique, the solution depends continuously on the data (e.g. it is stable) ...
aren't eistert's user avatar
6 votes
2 answers
550 views

How does GPT-based language model like ChatGPT determine the n-th letter of a word?

I understand that GPT models process input text by converting words into tokens and then embedding vectors and do not process them letter by letter. Given this approach, I am curious to know how a ...
Peyman's user avatar
  • 564
5 votes
2 answers
3k views

How is the next token predicted in transformers?

In the transformer (or GPT/decoder only), at the end of the decoder blocks but before the final linear layer you have X vectors (for the X tokens at the input of the decoder). We then want to compute ...
Miguel Carvalho's user avatar
4 votes
1 answer
145 views

What puts the "chat" in a system like ChatGPT?

So I read Wolfram's What Is ChatGPT Doing … and Why Does It Work? but it left one really big question in my mind. His summary [if it could be called that!] really emphasizes that the core model is ...
natevw's user avatar
  • 141
0 votes
2 answers
841 views

Is the GPT-4 for text the same model that can input and output images?

Currently, the published GPT-4 can input and output text. A version of GPT-4 that can input and output text and images exists, according to the technical report, but is not yet publicly available. I ...
Volker Siegel's user avatar
0 votes
1 answer
390 views

What is the difference between T5 and T0 models?

What is the difference between T5 and T0 models? I had read that T0 is T5 + LM. But as I know T5 uses encoder-decoder model like BART but BART can be used as LM so that's mean that T5 has a LM ...
prostak's user avatar
  • 103
1 vote
0 answers
43 views

Can I reduce computation by only predicting response tokens in a transformer and still get the same gradients?

I have been looking at the source code of the Stanford Alpaca model and I believe that during inference, the whole instruction + response data is fed into the model normally. Then the instruction part ...
Tianchen Zheng's user avatar
2 votes
0 answers
359 views

What's the most efficient way of performing batched training of Causal Language Models?

I have seen a number of ways to train (yes, train, not fine-tune) these models efficiently with batches. I will illustrate these techniques with the following example dataset and context window: ...
thesofakillers's user avatar
1 vote
1 answer
194 views

If we prompt a large language model on a task, will its ability for other tasks be affected? How to recover?

For example, I guess that for some retrieval augmented LLMs, their generated contents may lack some creativity? Recent work has explored the inability of retrieval augmented methods to enhance the ...
Iris88's user avatar
  • 11
0 votes
1 answer
4k views

How do temperature and repetition penalty interfere?

I'm trying to demystify my understanding of various decoding parameters. Building on our understanding of temperature, how does the repetition penalty interfere with temperature? For example, does ...
Corbin's user avatar
  • 103
1 vote
0 answers
89 views

Coding a conversational AI which remembers previous context

I am trying to code a proper conversational AI which remembers previous context and answers accordingly (something like a micro ChatGPT). Additionally I want the AI to work on a custom knowledge base ...
JAYDEEP GHOSE's user avatar
1 vote
0 answers
765 views

How to best implement Accelerate for vary large models

This was pointed out to me: https://huggingface.co/blog/accelerate-large-models and seems it could be a great resource, but it is broken down into steps as a tutorial vs as out of the box solution ...
jmhead's user avatar
  • 111
1 vote
1 answer
154 views

What background should I have before starting to fine tune a Large Language Model?

I want to know what things I should be learning before trying to fine-tune or for that matter working with a large language model. In my case, I am trying to fine-tune bloom (https://huggingface.co/...
kelly's user avatar
  • 23
2 votes
1 answer
150 views

Would a transformer trained on highly specific material be as usable as a commercial product like ChatGPT?

Soft question here. I was recently learning a bit about how it is feasible to train a transformer on a personal computer like an M1 Mac. I have been told that the model could have 1-3 million ...
Julius Hamilton's user avatar
16 votes
2 answers
7k views

Why does ChatGPT not give the answer text all at once?

When ChatGPT is generating an answer to my question, it generates it word by word. So I actually have to wait until I get the final answer. Is this just for show? Or is it really real-time generating ...
Sander van den Oord's user avatar
4 votes
2 answers
486 views

What makes reproducing a model like GPT3/GPT3.5/ChatGPT difficult?

Is it difficult for other companies to train a model similar to ChatGPT, and what makes it difficult? What is challenging about reproducing the results obtained by OpenAI with ChatGPT/GPT3.5? Would it ...
Robin van Hoorn's user avatar
1 vote
0 answers
85 views

How can i create a new language model for language other than english?

I have large set of corpus for all literature in 'Tamil' language, i am trying to create a document retrieval engine through simple natural language. Since the corpus is huge, its hard to do a ...
Thiyaga B's user avatar
  • 111
50 votes
3 answers
23k views

Was ChatGPT trained on Stack Overflow data?

Has ChatGPT used highly rated and upvoted questions/answers from Stack Overflow in its training data? For me it makes complete sense to take answers that have upwards of 100 upvotes and include them ...
Nicolas Zein's user avatar
1 vote
1 answer
609 views

For specific tasks, is it better to fine-tune models on examples or just use prompting with the context of the task?

These days large language models cover a vast amount of topics and information, but I wanted to understand: For specific tasks, is it better to fine-tune models on examples or just use prompting with ...
Imran Q's user avatar
  • 113
6 votes
1 answer
2k views

How was ChatGPT trained?

I know that large language models like GPT-3 are trained simply to continue pieces of text that have been scraped from the web. But how was ChatGPT trained, which, while also having a good ...
HelloGoodbye's user avatar
6 votes
1 answer
746 views

How can a language model keep track of the provenance of the main knowledge/sources used to generate a given output?

One of the main criticisms against the use of ChatGPT on Stack Exchange is that it doesn't attribute the main knowledge/sources used to generate a given output. How can a language model keep track of ...
Franck Dernoncourt's user avatar
9 votes
1 answer
1k views

What causes ChatGPT to generate responses that refer to itself as a bot or LM?

ChatGPT occasionally generates responses to prompts that refer to itself as a "bot" or "language model." For instance, when given a certain input (the first paragraph of this ...
Obie 2.0's user avatar
  • 192
24 votes
4 answers
12k views

How does ChatGPT know math?

ChatGPT is a language model. As far as I know and If I'm not wrong, it gets text as tokens and word embeddings. So, how can it do math? For example, I asked: ME: Which one is bigger 5 or 9. ChatGPT: ...
Peyman's user avatar
  • 564
1 vote
0 answers
23 views

How can (pretrained) language models actively seek additional training data - possibly reference request?

I am reading the paper "Large Language Models Can Self-Improve" https://arxiv.org/abs/2210.11610 in which the authors consider that LLM can generate Chain-of-Thoughts sequences and even ...
TomR's user avatar
  • 843
1 vote
0 answers
27 views

Are custom tokens better than punctuation pseudo-tokens for LLMs?

I've seen two approaches for introducing custom tokens for transfer learning with large language models like Bert or GPT3. Some approaches introduce new tokens into the vocabulary and learn embeddings ...
A Tyshka's user avatar
  • 111