Questions tagged [language-model]
For questions related to the concept of a language model, which is a probability distribution over sequences of words (for example, of a natural language, such as English).
73
questions
1
vote
1
answer
44
views
NLP "small" model to improve "big" model
When training the model for NLP is it important to get rid of data which has "bad semantic" for learning process?
My plan is to create a "small model" which can decide whether data ...
- 125
0
votes
0
answers
30
views
How can Knowledge Graphs be Integrated with Language Models for Semantic Search?
I am exploring the incorporate knowledge graphs (KGs) with language models.
I understand that KGs can provide structured understanding of entities and their relationships which can be crucial for ...
- 293
1
vote
1
answer
65
views
What is the lowest possible loss for a language model?
Example: Suppose a character-level language model (three input letters to predict the next one), trained on a dataset which contains three instances of the sequence ...
- 179
0
votes
1
answer
32
views
Setting number of rows returned by vector stores
When using vector stores like pinecone or Faiss from langchain, is it possible to set the number of records returned based on similarity search?
For example, consider the following code, is there a ...
2
votes
0
answers
32
views
How do I balance context and history when creating prompts for LLM's?
A conversation through the OpenAI API looks something like this
...
- 121
1
vote
2
answers
48
views
Data preparation for NLP model
I have data from our ticketing system. Currently using OpenNLP to create different models.
For simplicity I have a 10k ticket's text as category final queue of the ticket.
My questions:
Is it ...
- 125
0
votes
0
answers
35
views
Seeking Advice on Local AI Model for Analyzing Personal Files
I'm embarking on a project to develop an application for personal use that can assist me in retrieving specific information from my files. I'd like to be able to ask questions like "What was my ...
- 101
0
votes
1
answer
47
views
Why do current language models no longer generate to long or short texts?
One of the biggest strengths of ChatGPT is that it generates fitting text with respect to the input query. It usually stays on topic, anwers the question completely and especially does not start ...
- 1
1
vote
1
answer
68
views
How to train a language model to write poem?
I am using some of the LLM applications, and all of them are better at provide formal, steady conversations, rather than writing texts with styles. So I wonder whether it is possible to train a LLM ...
- 11
3
votes
1
answer
64
views
How can I improve this toy Graph Neural Network Generative Language model [closed]
Background
I'm an undergraduate student with research interests in a field of physics that has significant overlap with graph theory, and a functioning knowledge of how simple neural nets work and how ...
0
votes
0
answers
29
views
Create samples out of documents for Causal Language Modelling
I want to create an input source for Causal Language model using Llama 2 model in hugging face. I have a set of documents which are scraped from a specific website and want to fine-tune on them. Each ...
- 1
0
votes
0
answers
18
views
Small language model vocabulary correction
I plan to train a small language model (less than 4B parameters) that can run on x86 and handle vocabulary correction such as:
...
0
votes
0
answers
246
views
Optimal Quantity of Training Data for Fine-Tuning an LLM: Is Bigger Always Better?
I am currently working on fine-tuning an LLM for a specific task, and I am trying to determine the optimal size for my training dataset. Intuitively, one might think that the more data, the better. ...
- 554
0
votes
1
answer
76
views
Confusion About Triangle Mask in Transformer Decoder
I have some confusion about the implementation of the triangle mask in the transformer decoder. I understand the reasoning for the mask, it prevents the network from 'cheating' by looking ahead at the ...
- 1
1
vote
1
answer
96
views
Is there a limitation to the amount of data that a genAI model could be trained upon?
My friend says that genAI would become more human like, and perhaps even smarter than humans if it were simply trained on more and more data. I say that this would overtrain the models, and we would ...
0
votes
0
answers
93
views
A technique to show what tokens are relatively predicted by an LLM
I’m picturing a technique where you can see what an LLM is likely to respond with, which updates in real time.
It’s a bit trippy, but it’s like GitHub Copilot, in that there is predicted text while ...
- 133
0
votes
0
answers
254
views
How big the context can be using HuggingFace models?
I'm new on AI, Neural Networks, ChatBots and all this ecosystem. I'm trying to use a classical example of pre-trained models, more specifically ...
- 101
0
votes
0
answers
12
views
When running reinforcement learning on a language model, how do you prevent the model from just minimizing its logits?
Suppose you have a pre-trained autoregressive language model, and some cost function C mapping strings to numbers. Lower costs mean a given generated string is "better". Pass the model some ...
- 252
1
vote
1
answer
61
views
Is the problem of Language Modelling a Well-Posed Learning Problem?
Hadamard defines (Well-posed problem (Wikipedia)) a well-posed problem as one for which:
a solution exists,
the solution is unique,
the solution depends continuously on the data (e.g. it is stable)
...
6
votes
2
answers
390
views
How does GPT-based language model like ChatGPT determine the n-th letter of a word?
I understand that GPT models process input text by converting words into tokens and then embedding vectors and do not process them letter by letter. Given this approach, I am curious to know how a ...
- 554
5
votes
2
answers
2k
views
How is the next token predicted in transformers?
In the transformer (or GPT/decoder only), at the end of the decoder blocks but before the final linear layer you have X vectors (for the X tokens at the input of the decoder). We then want to compute ...
4
votes
1
answer
123
views
What puts the "chat" in a system like ChatGPT?
So I read Wolfram's What Is ChatGPT Doing … and Why Does It Work? but it left one really big question in my mind. His summary [if it could be called that!] really emphasizes that the core model is ...
- 141
0
votes
2
answers
753
views
Is the GPT-4 for text the same model that can input and output images?
Currently, the published GPT-4 can input and output text.
A version of GPT-4 that can input and output text and images exists, according to the technical report, but is not yet publicly available.
I ...
- 657
0
votes
0
answers
62
views
Does GPT not need to compress its training data?
In his recent short pamphlet on GPT, Stephan Wolfram says
... the 'size of the network' that seems to work well is ... comparable to the 'size of the training data'. ...
in this representation it ...
- 103
0
votes
1
answer
296
views
What is the difference between T5 and T0 models?
What is the difference between T5 and T0 models? I had read that T0 is T5 + LM. But as I know T5 uses encoder-decoder model like BART but BART can be used as LM so that's mean that T5 has a LM ...
- 103
0
votes
0
answers
123
views
Can an LLM such as GPT learn from a sentence prediction that is close in meaning but drastically different in wording?
Consider this example: husband cheats on wife, AI predicts that wife gets angry. AI predicts that wife may either scream or throw things, with screaming a higher probability. Turns out in reality ...
- 235
1
vote
0
answers
37
views
Can I reduce computation by only predicting response tokens in a transformer and still get the same gradients?
I have been looking at the source code of the Stanford Alpaca model and I believe that during inference, the whole instruction + response data is fed into the model normally. Then the instruction part ...
2
votes
0
answers
232
views
What's the most efficient way of performing batched training of Causal Language Models?
I have seen a number of ways to train (yes, train, not fine-tune) these models efficiently with batches. I will illustrate these techniques with the following example dataset and context window:
...
- 309
1
vote
1
answer
118
views
If we prompt a large language model on a task, will its ability for other tasks be affected? How to recover?
For example, I guess that for some retrieval augmented LLMs, their generated contents may lack some creativity? Recent work has explored the inability of retrieval augmented methods to enhance the ...
- 11
0
votes
0
answers
26
views
How are the softmax normalized weights in ELMo actually learned and computed?
I was reading the ELMo paper, and they speak of task-specific representations of words (or tokens generally speaking) by using the following equation: $ELMo_{k}^{task} = \gamma^{task}\sum_{0}^{L}{s_{j}...
- 1
0
votes
0
answers
9
views
Why is avoiding normalized models a practical solution for reducing the complexity in NNLM?
In the paper Efficient Estimation of Word Representations in Vector Space, the authors say that "avoiding normalized models completely by using models that are not normalized during training"...
- 1
0
votes
1
answer
3k
views
How do temperature and repetition penalty interfere?
I'm trying to demystify my understanding of various decoding parameters. Building on our understanding of temperature, how does the repetition penalty interfere with temperature?
For example, does ...
- 103
1
vote
0
answers
88
views
Coding a conversational AI which remembers previous context
I am trying to code a proper conversational AI which remembers previous context and answers accordingly (something like a micro ChatGPT). Additionally I want the AI to work on a custom knowledge base ...
1
vote
0
answers
677
views
How to best implement Accelerate for vary large models
This was pointed out to me: https://huggingface.co/blog/accelerate-large-models
and seems it could be a great resource, but it is broken down into steps as a tutorial vs as out of the box solution ...
- 111
1
vote
1
answer
147
views
What background should I have before starting to fine tune a Large Language Model?
I want to know what things I should be learning before trying to fine-tune or for that matter working with a large language model.
In my case, I am trying to fine-tune bloom (https://huggingface.co/...
- 23
0
votes
0
answers
17
views
Text Classification Model unable to learn
I am trying to build a text classification model. When I train the model it is unable to improve accuracy and at some point accuracy even decreases and loss increases.
I have researched for possible ...
- 1
2
votes
1
answer
135
views
Would a transformer trained on highly specific material be as usable as a commercial product like ChatGPT?
Soft question here.
I was recently learning a bit about how it is feasible to train a transformer on a personal computer like an M1 Mac. I have been told that the model could have 1-3 million ...
- 133
13
votes
2
answers
6k
views
Why does ChatGPT not give the answer text all at once?
When ChatGPT is generating an answer to my question, it generates it word by word.
So I actually have to wait until I get the final answer.
Is this just for show?
Or is it really real-time generating ...
4
votes
2
answers
472
views
What makes reproducing a model like GPT3/GPT3.5/ChatGPT difficult?
Is it difficult for other companies to train a model similar to ChatGPT, and what makes it difficult? What is challenging about reproducing the results obtained by OpenAI with ChatGPT/GPT3.5? Would it ...
- 2,083
1
vote
0
answers
71
views
How can i create a new language model for language other than english?
I have large set of corpus for all literature in 'Tamil' language, i am trying to create a document retrieval engine through simple natural language.
Since the corpus is huge, its hard to do a ...
- 111
49
votes
3
answers
22k
views
Was ChatGPT trained on Stack Overflow data?
Has ChatGPT used highly rated and upvoted questions/answers from Stack Overflow in its training data?
For me it makes complete sense to take answers that have upwards of 100 upvotes and include them ...
- 591
1
vote
1
answer
524
views
For specific tasks, is it better to fine-tune models on examples or just use prompting with the context of the task?
These days large language models cover a vast amount of topics and information, but I wanted to understand: For specific tasks, is it better to fine-tune models on examples or just use prompting with ...
- 113
6
votes
1
answer
2k
views
How was ChatGPT trained?
I know that large language models like GPT-3 are trained simply to continue pieces of text that have been scraped from the web. But how was ChatGPT trained, which, while also having a good ...
- 313
6
votes
1
answer
622
views
How can a language model keep track of the provenance of the main knowledge/sources used to generate a given output?
One of the main criticisms against the use of ChatGPT on Stack Exchange is that it doesn't attribute the main knowledge/sources used to generate a given output. How can a language model keep track of ...
- 2,804
9
votes
1
answer
1k
views
What causes ChatGPT to generate responses that refer to itself as a bot or LM?
ChatGPT occasionally generates responses to prompts that refer to itself as a "bot" or "language model."
For instance, when given a certain input (the first paragraph of this ...
- 193
24
votes
4
answers
10k
views
How does ChatGPT know math?
ChatGPT is a language model. As far as I know and If I'm not wrong, it gets text as tokens and word embeddings. So, how can it do math? For example, I asked:
ME: Which one is bigger 5 or 9.
ChatGPT: ...
- 554
1
vote
0
answers
22
views
How can (pretrained) language models actively seek additional training data - possibly reference request?
I am reading the paper "Large Language Models Can Self-Improve" https://arxiv.org/abs/2210.11610 in which the authors consider that LLM can generate Chain-of-Thoughts sequences and even ...
- 833
1
vote
0
answers
26
views
Are custom tokens better than punctuation pseudo-tokens for LLMs?
I've seen two approaches for introducing custom tokens for transfer learning with large language models like Bert or GPT3. Some approaches introduce new tokens into the vocabulary and learn embeddings ...
- 111
7
votes
2
answers
3k
views
Why can't language models, like GPT-3, continuously learn once trained?
GPT-3 has a prompt limit of about ~2048 "tokens", which corresponds to about 4 characters in text. If my understanding is correct, a deep neural network is not learning after it is trained ...
- 355
1
vote
1
answer
805
views
Papers on Prompt Engineering
I am into AI in general and NLP in particular. Besides, I have a background in philosophy, and the new LLMs like GPT-3 seem to have exciting capabilities. I want to study prompt engineering (for ...
- 123