Questions tagged [language-model]

For questions related to the concept of a language model, which is a probability distribution over sequences of words (for example, of a natural language, such as English).

Filter by
Sorted by
Tagged with
1 vote
1 answer
44 views

NLP "small" model to improve "big" model

When training the model for NLP is it important to get rid of data which has "bad semantic" for learning process? My plan is to create a "small model" which can decide whether data ...
Milkmaid's user avatar
  • 125
0 votes
0 answers
30 views

How can Knowledge Graphs be Integrated with Language Models for Semantic Search?

I am exploring the incorporate knowledge graphs (KGs) with language models. I understand that KGs can provide structured understanding of entities and their relationships which can be crucial for ...
Exploring's user avatar
  • 293
1 vote
1 answer
65 views

What is the lowest possible loss for a language model?

Example: Suppose a character-level language model (three input letters to predict the next one), trained on a dataset which contains three instances of the sequence ...
ViniciusArruda's user avatar
0 votes
1 answer
32 views

Setting number of rows returned by vector stores

When using vector stores like pinecone or Faiss from langchain, is it possible to set the number of records returned based on similarity search? For example, consider the following code, is there a ...
Karl 17302's user avatar
2 votes
0 answers
32 views

How do I balance context and history when creating prompts for LLM's?

A conversation through the OpenAI API looks something like this ...
Ian Purton's user avatar
1 vote
2 answers
48 views

Data preparation for NLP model

I have data from our ticketing system. Currently using OpenNLP to create different models. For simplicity I have a 10k ticket's text as category final queue of the ticket. My questions: Is it ...
Milkmaid's user avatar
  • 125
0 votes
0 answers
35 views

Seeking Advice on Local AI Model for Analyzing Personal Files

I'm embarking on a project to develop an application for personal use that can assist me in retrieving specific information from my files. I'd like to be able to ask questions like "What was my ...
NoNam4's user avatar
  • 101
0 votes
1 answer
47 views

Why do current language models no longer generate to long or short texts?

One of the biggest strengths of ChatGPT is that it generates fitting text with respect to the input query. It usually stays on topic, anwers the question completely and especially does not start ...
Ricu's user avatar
  • 1
1 vote
1 answer
68 views

How to train a language model to write poem?

I am using some of the LLM applications, and all of them are better at provide formal, steady conversations, rather than writing texts with styles. So I wonder whether it is possible to train a LLM ...
Cabbage's user avatar
  • 11
3 votes
1 answer
64 views

How can I improve this toy Graph Neural Network Generative Language model [closed]

Background I'm an undergraduate student with research interests in a field of physics that has significant overlap with graph theory, and a functioning knowledge of how simple neural nets work and how ...
MomentumEigenstate's user avatar
0 votes
0 answers
29 views

Create samples out of documents for Causal Language Modelling

I want to create an input source for Causal Language model using Llama 2 model in hugging face. I have a set of documents which are scraped from a specific website and want to fine-tune on them. Each ...
Dimits's user avatar
  • 1
0 votes
0 answers
18 views

Small language model vocabulary correction

I plan to train a small language model (less than 4B parameters) that can run on x86 and handle vocabulary correction such as: ...
Jeff Brower's user avatar
0 votes
0 answers
246 views

Optimal Quantity of Training Data for Fine-Tuning an LLM: Is Bigger Always Better?

I am currently working on fine-tuning an LLM for a specific task, and I am trying to determine the optimal size for my training dataset. Intuitively, one might think that the more data, the better. ...
Peyman's user avatar
  • 554
0 votes
1 answer
76 views

Confusion About Triangle Mask in Transformer Decoder

I have some confusion about the implementation of the triangle mask in the transformer decoder. I understand the reasoning for the mask, it prevents the network from 'cheating' by looking ahead at the ...
new2java's user avatar
1 vote
1 answer
96 views

Is there a limitation to the amount of data that a genAI model could be trained upon?

My friend says that genAI would become more human like, and perhaps even smarter than humans if it were simply trained on more and more data. I say that this would overtrain the models, and we would ...
tryst with freedom's user avatar
0 votes
0 answers
93 views

A technique to show what tokens are relatively predicted by an LLM

I’m picturing a technique where you can see what an LLM is likely to respond with, which updates in real time. It’s a bit trippy, but it’s like GitHub Copilot, in that there is predicted text while ...
hmltn's user avatar
  • 133
0 votes
0 answers
254 views

How big the context can be using HuggingFace models?

I'm new on AI, Neural Networks, ChatBots and all this ecosystem. I'm trying to use a classical example of pre-trained models, more specifically ...
Magno C's user avatar
  • 101
0 votes
0 answers
12 views

When running reinforcement learning on a language model, how do you prevent the model from just minimizing its logits?

Suppose you have a pre-trained autoregressive language model, and some cost function C mapping strings to numbers. Lower costs mean a given generated string is "better". Pass the model some ...
Jack M's user avatar
  • 252
1 vote
1 answer
61 views

Is the problem of Language Modelling a Well-Posed Learning Problem?

Hadamard defines (Well-posed problem (Wikipedia)) a well-posed problem as one for which: a solution exists, the solution is unique, the solution depends continuously on the data (e.g. it is stable) ...
aren't eistert's user avatar
6 votes
2 answers
390 views

How does GPT-based language model like ChatGPT determine the n-th letter of a word?

I understand that GPT models process input text by converting words into tokens and then embedding vectors and do not process them letter by letter. Given this approach, I am curious to know how a ...
Peyman's user avatar
  • 554
5 votes
2 answers
2k views

How is the next token predicted in transformers?

In the transformer (or GPT/decoder only), at the end of the decoder blocks but before the final linear layer you have X vectors (for the X tokens at the input of the decoder). We then want to compute ...
Miguel Carvalho's user avatar
4 votes
1 answer
123 views

What puts the "chat" in a system like ChatGPT?

So I read Wolfram's What Is ChatGPT Doing … and Why Does It Work? but it left one really big question in my mind. His summary [if it could be called that!] really emphasizes that the core model is ...
natevw's user avatar
  • 141
0 votes
2 answers
753 views

Is the GPT-4 for text the same model that can input and output images?

Currently, the published GPT-4 can input and output text. A version of GPT-4 that can input and output text and images exists, according to the technical report, but is not yet publicly available. I ...
Volker Siegel's user avatar
0 votes
0 answers
62 views

Does GPT not need to compress its training data?

In his recent short pamphlet on GPT, Stephan Wolfram says ... the 'size of the network' that seems to work well is ... comparable to the 'size of the training data'. ... in this representation it ...
orome's user avatar
  • 103
0 votes
1 answer
296 views

What is the difference between T5 and T0 models?

What is the difference between T5 and T0 models? I had read that T0 is T5 + LM. But as I know T5 uses encoder-decoder model like BART but BART can be used as LM so that's mean that T5 has a LM ...
prostak's user avatar
  • 103
0 votes
0 answers
123 views

Can an LLM such as GPT learn from a sentence prediction that is close in meaning but drastically different in wording?

Consider this example: husband cheats on wife, AI predicts that wife gets angry. AI predicts that wife may either scream or throw things, with screaming a higher probability. Turns out in reality ...
Yan King Yin's user avatar
1 vote
0 answers
37 views

Can I reduce computation by only predicting response tokens in a transformer and still get the same gradients?

I have been looking at the source code of the Stanford Alpaca model and I believe that during inference, the whole instruction + response data is fed into the model normally. Then the instruction part ...
Tianchen Zheng's user avatar
2 votes
0 answers
232 views

What's the most efficient way of performing batched training of Causal Language Models?

I have seen a number of ways to train (yes, train, not fine-tune) these models efficiently with batches. I will illustrate these techniques with the following example dataset and context window: ...
thesofakillers's user avatar
1 vote
1 answer
118 views

If we prompt a large language model on a task, will its ability for other tasks be affected? How to recover?

For example, I guess that for some retrieval augmented LLMs, their generated contents may lack some creativity? Recent work has explored the inability of retrieval augmented methods to enhance the ...
Iris88's user avatar
  • 11
0 votes
0 answers
26 views

How are the softmax normalized weights in ELMo actually learned and computed?

I was reading the ELMo paper, and they speak of task-specific representations of words (or tokens generally speaking) by using the following equation: $ELMo_{k}^{task} = \gamma^{task}\sum_{0}^{L}{s_{j}...
Propr's user avatar
  • 1
0 votes
0 answers
9 views

Why is avoiding normalized models a practical solution for reducing the complexity in NNLM?

In the paper Efficient Estimation of Word Representations in Vector Space, the authors say that "avoiding normalized models completely by using models that are not normalized during training"...
Propr's user avatar
  • 1
0 votes
1 answer
3k views

How do temperature and repetition penalty interfere?

I'm trying to demystify my understanding of various decoding parameters. Building on our understanding of temperature, how does the repetition penalty interfere with temperature? For example, does ...
Corbin's user avatar
  • 103
1 vote
0 answers
88 views

Coding a conversational AI which remembers previous context

I am trying to code a proper conversational AI which remembers previous context and answers accordingly (something like a micro ChatGPT). Additionally I want the AI to work on a custom knowledge base ...
JAYDEEP GHOSE's user avatar
1 vote
0 answers
677 views

How to best implement Accelerate for vary large models

This was pointed out to me: https://huggingface.co/blog/accelerate-large-models and seems it could be a great resource, but it is broken down into steps as a tutorial vs as out of the box solution ...
jmhead's user avatar
  • 111
1 vote
1 answer
147 views

What background should I have before starting to fine tune a Large Language Model?

I want to know what things I should be learning before trying to fine-tune or for that matter working with a large language model. In my case, I am trying to fine-tune bloom (https://huggingface.co/...
kelly's user avatar
  • 23
0 votes
0 answers
17 views

Text Classification Model unable to learn

I am trying to build a text classification model. When I train the model it is unable to improve accuracy and at some point accuracy even decreases and loss increases. I have researched for possible ...
javi11br's user avatar
2 votes
1 answer
135 views

Would a transformer trained on highly specific material be as usable as a commercial product like ChatGPT?

Soft question here. I was recently learning a bit about how it is feasible to train a transformer on a personal computer like an M1 Mac. I have been told that the model could have 1-3 million ...
hmltn's user avatar
  • 133
13 votes
2 answers
6k views

Why does ChatGPT not give the answer text all at once?

When ChatGPT is generating an answer to my question, it generates it word by word. So I actually have to wait until I get the final answer. Is this just for show? Or is it really real-time generating ...
Sander van den Oord's user avatar
4 votes
2 answers
472 views

What makes reproducing a model like GPT3/GPT3.5/ChatGPT difficult?

Is it difficult for other companies to train a model similar to ChatGPT, and what makes it difficult? What is challenging about reproducing the results obtained by OpenAI with ChatGPT/GPT3.5? Would it ...
Robin van Hoorn's user avatar
1 vote
0 answers
71 views

How can i create a new language model for language other than english?

I have large set of corpus for all literature in 'Tamil' language, i am trying to create a document retrieval engine through simple natural language. Since the corpus is huge, its hard to do a ...
Thiyaga B's user avatar
  • 111
49 votes
3 answers
22k views

Was ChatGPT trained on Stack Overflow data?

Has ChatGPT used highly rated and upvoted questions/answers from Stack Overflow in its training data? For me it makes complete sense to take answers that have upwards of 100 upvotes and include them ...
Nicolas Zein's user avatar
1 vote
1 answer
524 views

For specific tasks, is it better to fine-tune models on examples or just use prompting with the context of the task?

These days large language models cover a vast amount of topics and information, but I wanted to understand: For specific tasks, is it better to fine-tune models on examples or just use prompting with ...
Imran Q's user avatar
  • 113
6 votes
1 answer
2k views

How was ChatGPT trained?

I know that large language models like GPT-3 are trained simply to continue pieces of text that have been scraped from the web. But how was ChatGPT trained, which, while also having a good ...
HelloGoodbye's user avatar
6 votes
1 answer
622 views

How can a language model keep track of the provenance of the main knowledge/sources used to generate a given output?

One of the main criticisms against the use of ChatGPT on Stack Exchange is that it doesn't attribute the main knowledge/sources used to generate a given output. How can a language model keep track of ...
Franck Dernoncourt's user avatar
9 votes
1 answer
1k views

What causes ChatGPT to generate responses that refer to itself as a bot or LM?

ChatGPT occasionally generates responses to prompts that refer to itself as a "bot" or "language model." For instance, when given a certain input (the first paragraph of this ...
Obie 2.0's user avatar
  • 193
24 votes
4 answers
10k views

How does ChatGPT know math?

ChatGPT is a language model. As far as I know and If I'm not wrong, it gets text as tokens and word embeddings. So, how can it do math? For example, I asked: ME: Which one is bigger 5 or 9. ChatGPT: ...
Peyman's user avatar
  • 554
1 vote
0 answers
22 views

How can (pretrained) language models actively seek additional training data - possibly reference request?

I am reading the paper "Large Language Models Can Self-Improve" https://arxiv.org/abs/2210.11610 in which the authors consider that LLM can generate Chain-of-Thoughts sequences and even ...
TomR's user avatar
  • 833
1 vote
0 answers
26 views

Are custom tokens better than punctuation pseudo-tokens for LLMs?

I've seen two approaches for introducing custom tokens for transfer learning with large language models like Bert or GPT3. Some approaches introduce new tokens into the vocabulary and learn embeddings ...
A Tyshka's user avatar
  • 111
7 votes
2 answers
3k views

Why can't language models, like GPT-3, continuously learn once trained?

GPT-3 has a prompt limit of about ~2048 "tokens", which corresponds to about 4 characters in text. If my understanding is correct, a deep neural network is not learning after it is trained ...
MaiaVictor's user avatar
1 vote
1 answer
805 views

Papers on Prompt Engineering

I am into AI in general and NLP in particular. Besides, I have a background in philosophy, and the new LLMs like GPT-3 seem to have exciting capabilities. I want to study prompt engineering (for ...
Mehdi Abbassi's user avatar