Questions tagged [language-model]
For questions related to the concept of a language model, which is a probability distribution over sequences of words (for example, of a natural language, such as English).
46
questions
0
votes
1
answer
28
views
If we prompt a large language model on a task, will its ability for other tasks be affected? How to recover?
For example, I guess that for some retrieval augmented LLMs, their generated contents may lack some creativity? Recent work has explored the inability of retrieval augmented methods to enhance the ...
- 1
0
votes
0
answers
7
views
How are the softmax normalized weights in ELMo actually learned and computed?
I was reading the ELMo paper, and they speak of task-specific representations of words (or tokens generally speaking) by using the following equation: $ELMo_{k}^{task} = \gamma^{task}\sum_{0}^{L}{s_{j}...
- 1
0
votes
0
answers
8
views
Why is avoiding normalized models a practical solution for reducing the complexity in NNLM?
In the paper Efficient Estimation of Word Representations in Vector Space, the authors say that "avoiding normalized models completely by using models that are not normalized during training"...
- 1
0
votes
1
answer
39
views
How do temperature and repetition penalty interfere?
I'm trying to demystify my understanding of various decoding parameters. Building on our understanding of temperature, how does the repetition penalty interfere with temperature?
For example, does ...
- 101
1
vote
0
answers
67
views
Coding a conversational AI which remembers previous context
I am trying to code a proper conversational AI which remembers previous context and answers accordingly (something like a micro ChatGPT). Additionally I want the AI to work on a custom knowledge base ...
0
votes
0
answers
52
views
How to best implement Accelerate for vary large models
This was pointed out to me: https://huggingface.co/blog/accelerate-large-models
and seems it could be a great resource, but it is broken down into steps as a tutorial vs as out of the box solution ...
- 101
1
vote
1
answer
87
views
What background should I have before starting to fine tune a Large Language Model?
I want to know what things I should be learning before trying to fine-tune or for that matter working with a large language model.
In my case, I am trying to fine-tune bloom (https://huggingface.co/...
- 23
0
votes
0
answers
15
views
Text Classification Model unable to learn
I am trying to build a text classification model. When I train the model it is unable to improve accuracy and at some point accuracy even decreases and loss increases.
I have researched for possible ...
- 1
2
votes
1
answer
72
views
Would a transformer trained on highly specific material be as usable as a commercial product like ChatGPT?
Soft question here.
I was recently learning a bit about how it is feasible to train a transformer on a personal computer like an M1 Mac. I have been told that the model could have 1-3 million ...
- 119
10
votes
2
answers
1k
views
Why does ChatGPT not give the answer text all at once?
When ChatGPT is generating an answer to my question, it generates it word by word.
So I actually have to wait until I get the final answer.
Is this just for show?
Or is it really real-time generating ...
4
votes
2
answers
322
views
What makes reproducing a model like GPT3/GPT3.5/ChatGPT difficult?
Is it difficult for other companies to train a model similar to ChatGPT, and what makes it difficult? What is challenging about reproducing the results obtained by OpenAI with ChatGPT/GPT3.5? Would it ...
- 808
1
vote
0
answers
46
views
How can i create a new language model for language other than english?
I have large set of corpus for all literature in 'Tamil' language, i am trying to create a document retrieval engine through simple natural language.
Since the corpus is huge, its hard to do a ...
- 111
1
vote
1
answer
154
views
For specific tasks, is it better to fine-tune models on examples or just use prompting with the context of the task?
These days large language models cover a vast amount of topics and information, but I wanted to understand: For specific tasks, is it better to fine-tune models on examples or just use prompting with ...
- 113
6
votes
1
answer
2k
views
How was ChatGPT trained?
I know that large language models like GPT-3 are trained simply to continue pieces of text that have been scraped from the web. But how was ChatGPT trained, which, while also having a good ...
- 313
1
vote
1
answer
164
views
How can a language model keep track of the provenance of the main knowledge/sources used to generate a given output?
One of the main criticisms against the use of ChatGPT on stack exchange is that it doesn't attribute the main knowledge/sources used to generate a given output. How can a language model keep track of ...
- 2,346
6
votes
1
answer
538
views
What causes ChatGPT to generate responses that refer to itself as a bot or LM?
ChatGPT occasionally generates responses to prompts that refer to itself as a "bot" or "language model."
For instance, when given a certain input (the first paragraph of this ...
- 163
17
votes
3
answers
6k
views
How does ChatGPT know math?
ChatGPT is a language model. As far as I know and If I'm not wrong, it gets text as tokens and word embeddings. So, how can it do math? For example, I asked:
ME: Which one is bigger 5 or 9.
ChatGPT: ...
- 322
1
vote
0
answers
12
views
How can (pretrained) language models actively seek additional training data - possibly reference request?
I am reading the paper "Large Language Models Can Self-Improve" https://arxiv.org/abs/2210.11610 in which the authors consider that LLM can generate Chain-of-Thoughts sequences and even ...
- 803
0
votes
0
answers
27
views
Instruction-based language model without neural network
What kind of technology could be used to develop an AI with the same use case as InstructGPT (i.e. generating text from an instruction) but without using neural networks?
Obviously the performance ...
user62498
0
votes
0
answers
10
views
Are custom tokens better than punctuation pseudo-tokens for LLMs?
I've seen two approaches for introducing custom tokens for transfer learning with large language models like Bert or GPT3. Some approaches introduce new tokens into the vocabulary and learn embeddings ...
- 101
6
votes
3
answers
2k
views
Why can't language models, like GPT-3, continuously learn once trained?
GPT-3 has a prompt limit of about ~2048 "tokens", which corresponds to about 4 characters in text. If my understanding is correct, a deep neural network is not learning after it is trained ...
- 173
0
votes
1
answer
256
views
Papers on Prompt Engineering
I am into AI in general and NLP in particular. Besides, I have a background in philosophy, and the new LLMs like GPT-3 seem to have exciting capabilities. I want to study prompt engineering (for ...
- 115
1
vote
0
answers
22
views
Which language model technique should be used with a low-resource language?
I am working with a language for which the amount of text available is relatively small (less than a billion words), what techniques exist? In particular, is it worth using a transformer or is it ...
- 11
0
votes
0
answers
54
views
BERT predictive distribution for non-masked tokens
In the BERT paper, the masking regime is described in the following way:
The training data generator chooses 15% of the token positions at random for prediction. If the i-th token is chosen, we ...
- 101
0
votes
2
answers
400
views
How to combine pretrained language models with additional feature set?
Are there any techniques to combine a feature set (other than the text itself) with pretrained language models.
Let's say I have a random NLP task that tries to predict a binary class label based on e....
- 101
0
votes
0
answers
13
views
What is the next-character perplexity of the PaLM model?
In the 2022 paper "PaLM: Scaling Language Modeling with Pathways", what is the bits-per-character perplexity of the resultant pre-trained model for next-word prediction?
- 101
13
votes
1
answer
12k
views
What is the "temperature" in the GPT models?
What does the temperature parameter mean when talking about the GPT models?
I know that a higher temperature value means more randomness, but I want to know how randomness is introduced.
Does ...
- 253
0
votes
1
answer
347
views
Why do language models produce different outputs for same prompt?
For conventional 'Neural Networks', the weights simply act as a transformation in highly multi-dimensional space; for a forward pass, the output is always the same since there is no stochastic ...
- 126
2
votes
1
answer
1k
views
What is input (and shape) to K/V/Q of self-attention of EACH Decoder block of Language-translation model Transformer's tokens during Inference?
Transformer model of the original Attention paper has a decoder unit that works differently during Inference than Tranining.
I'm trying to understand the shapes used during decoder (both self-...
- 171
2
votes
1
answer
63
views
Are training sequences for LMs sampled in an IID fashion?
If I understand correctly, when training language models, we take a document and then chunk the document into a sequences of k tokens. So if the document is of length 30 and k=10, then we'll have 20 ...
- 121
4
votes
2
answers
5k
views
What is the difference between a language model and a word embedding?
I am self-studying applications of deep learning on the NLP and machine translation.
I am confused about the concepts of "Language Model", "Word Embedding", "BLEU Score".
...
- 153
0
votes
1
answer
425
views
Are there any good alternatives to an LSTM language model for text generation?
I have a trained LSTM language model and want to use it to generate text. The standard approach for this seems to be:
Apply softmax function
Take a weighted random choice to determine the next word
...
1
vote
0
answers
9
views
When we translate a text from one language to another, how does the frequency of various POS tags change?
When we translate a text from one language to another, how does the frequency of various POS tags change?
So, let's say we have a text in English, with 10% nouns, 20% adjectives, 15% adverbs, 25% ...
- 11
1
vote
0
answers
143
views
How is input defined for a biaxial lstm network for generating music?
I am reading Composing Music With Recurrent Neural Networks by Daniel D. Johnson. But I am really confused about the input passed to this network. If we pass notes of music along the time axis, then ...
- 121
1
vote
1
answer
974
views
Is there a way to provide multiple masks to BERT in MLM task?
I'm facing a situation where I've to fetch probabilities from BERT MLM for multiple words in a single sentence.
...
- 317
1
vote
1
answer
52
views
Appropriate metric and approach for natural language generation for small sentences
I am trying to create a language generation model to generate very short sentences/words, like a rapper name generator. The sentences in my dataset are anywhere between 1 word and 15 words (3-155 ...
- 326
3
votes
1
answer
151
views
Fundamentally, what is a perfect language model?
Suppose that we want to generate a sentence made of words according to language $L$:
$$
W_1 W_2 \ldots W_n
$$
Question: What is the perfect language model?
I ask about perfect because I want to know ...
- 151
1
vote
1
answer
483
views
What are the main differences between a language model and a machine translation model?
What are the main differences between a language model and a machine translation model?
- 1,433
6
votes
1
answer
1k
views
What are pros and cons of Bi-LSTM as compared to LSTM?
What are the pros and cons of LSTM vs Bi-LSTM in language modelling? What was the need to introduce Bi-LSTM?
- 1,433
1
vote
0
answers
74
views
How to pad sequences during training for an encoder decoder model
I've got an encoder-decoder model for character level English language spelling correction, it is pretty basic stuff with a two LSTM encoder and another LSTM decoder.
However, up until now, I have ...
- 111
7
votes
1
answer
1k
views
How to use BERT as a multi-purpose conversational AI?
I'm looking to make an NLP model that can achieve a dual purpose. One purpose is that it can hold interesting conversations (conversational AI), and another being that it can do intent classification ...
- 323
4
votes
3
answers
183
views
What would be a good internal language for an AI?
For an AI to represent the world, it would be good if it could translate human sentences into something more precise.
We know, for example, that mathematics can be built up from set theory. So ...
- 2,156
4
votes
2
answers
154
views
Are embeddings in multi-lingual language models comparable across languages?
Facebook has just pushed out a bigger version of their multi-lingual language model XLM, called XLM-R. My question is: do these kind of multi-lingual models imply, or even ensure, that their ...
- 146
2
votes
1
answer
63
views
How do you build a language model to predict the contextual similarity between two documents?
How do you build a language model to predict the contextual similarity between two documents?
2
votes
0
answers
65
views
How can I generate a document from a single word using GPT or BERT?
I have a dataset of 100000 documents each labelled with a topic to it. I want to create a model such that, given a topic, the model can generate a document from it.
I came across language models GPT,...
- 161
5
votes
2
answers
2k
views
Where can I find pre-trained language models in English and German? [closed]
Where can I find (more) pre-trained language models? I am especially interested in neural network-based models for English and German.
I am aware only of Language Model on One Billion Word Benchmark ...
- 161