Questions tagged [language-model]

For questions related to the concept of a language model, which is a probability distribution over sequences of words (for example, of a natural language, such as English).

Filter by
Sorted by
Tagged with
0 votes
1 answer
28 views

If we prompt a large language model on a task, will its ability for other tasks be affected? How to recover?

For example, I guess that for some retrieval augmented LLMs, their generated contents may lack some creativity? Recent work has explored the inability of retrieval augmented methods to enhance the ...
  • 1
0 votes
0 answers
7 views

How are the softmax normalized weights in ELMo actually learned and computed?

I was reading the ELMo paper, and they speak of task-specific representations of words (or tokens generally speaking) by using the following equation: $ELMo_{k}^{task} = \gamma^{task}\sum_{0}^{L}{s_{j}...
  • 1
0 votes
0 answers
8 views

Why is avoiding normalized models a practical solution for reducing the complexity in NNLM?

In the paper Efficient Estimation of Word Representations in Vector Space, the authors say that "avoiding normalized models completely by using models that are not normalized during training"...
  • 1
0 votes
1 answer
39 views

How do temperature and repetition penalty interfere?

I'm trying to demystify my understanding of various decoding parameters. Building on our understanding of temperature, how does the repetition penalty interfere with temperature? For example, does ...
  • 101
1 vote
0 answers
67 views

Coding a conversational AI which remembers previous context

I am trying to code a proper conversational AI which remembers previous context and answers accordingly (something like a micro ChatGPT). Additionally I want the AI to work on a custom knowledge base ...
0 votes
0 answers
52 views

How to best implement Accelerate for vary large models

This was pointed out to me: https://huggingface.co/blog/accelerate-large-models and seems it could be a great resource, but it is broken down into steps as a tutorial vs as out of the box solution ...
  • 101
1 vote
1 answer
87 views

What background should I have before starting to fine tune a Large Language Model?

I want to know what things I should be learning before trying to fine-tune or for that matter working with a large language model. In my case, I am trying to fine-tune bloom (https://huggingface.co/...
  • 23
0 votes
0 answers
15 views

Text Classification Model unable to learn

I am trying to build a text classification model. When I train the model it is unable to improve accuracy and at some point accuracy even decreases and loss increases. I have researched for possible ...
2 votes
1 answer
72 views

Would a transformer trained on highly specific material be as usable as a commercial product like ChatGPT?

Soft question here. I was recently learning a bit about how it is feasible to train a transformer on a personal computer like an M1 Mac. I have been told that the model could have 1-3 million ...
10 votes
2 answers
1k views

Why does ChatGPT not give the answer text all at once?

When ChatGPT is generating an answer to my question, it generates it word by word. So I actually have to wait until I get the final answer. Is this just for show? Or is it really real-time generating ...
4 votes
2 answers
322 views

What makes reproducing a model like GPT3/GPT3.5/ChatGPT difficult?

Is it difficult for other companies to train a model similar to ChatGPT, and what makes it difficult? What is challenging about reproducing the results obtained by OpenAI with ChatGPT/GPT3.5? Would it ...
1 vote
0 answers
46 views

How can i create a new language model for language other than english?

I have large set of corpus for all literature in 'Tamil' language, i am trying to create a document retrieval engine through simple natural language. Since the corpus is huge, its hard to do a ...
  • 111
1 vote
1 answer
154 views

For specific tasks, is it better to fine-tune models on examples or just use prompting with the context of the task?

These days large language models cover a vast amount of topics and information, but I wanted to understand: For specific tasks, is it better to fine-tune models on examples or just use prompting with ...
  • 113
6 votes
1 answer
2k views

How was ChatGPT trained?

I know that large language models like GPT-3 are trained simply to continue pieces of text that have been scraped from the web. But how was ChatGPT trained, which, while also having a good ...
1 vote
1 answer
164 views

How can a language model keep track of the provenance of the main knowledge/sources used to generate a given output?

One of the main criticisms against the use of ChatGPT on stack exchange is that it doesn't attribute the main knowledge/sources used to generate a given output. How can a language model keep track of ...
6 votes
1 answer
538 views

What causes ChatGPT to generate responses that refer to itself as a bot or LM?

ChatGPT occasionally generates responses to prompts that refer to itself as a "bot" or "language model." For instance, when given a certain input (the first paragraph of this ...
  • 163
17 votes
3 answers
6k views

How does ChatGPT know math?

ChatGPT is a language model. As far as I know and If I'm not wrong, it gets text as tokens and word embeddings. So, how can it do math? For example, I asked: ME: Which one is bigger 5 or 9. ChatGPT: ...
  • 322
1 vote
0 answers
12 views

How can (pretrained) language models actively seek additional training data - possibly reference request?

I am reading the paper "Large Language Models Can Self-Improve" https://arxiv.org/abs/2210.11610 in which the authors consider that LLM can generate Chain-of-Thoughts sequences and even ...
  • 803
0 votes
0 answers
27 views

Instruction-based language model without neural network

What kind of technology could be used to develop an AI with the same use case as InstructGPT (i.e. generating text from an instruction) but without using neural networks? Obviously the performance ...
user avatar
0 votes
0 answers
10 views

Are custom tokens better than punctuation pseudo-tokens for LLMs?

I've seen two approaches for introducing custom tokens for transfer learning with large language models like Bert or GPT3. Some approaches introduce new tokens into the vocabulary and learn embeddings ...
  • 101
6 votes
3 answers
2k views

Why can't language models, like GPT-3, continuously learn once trained?

GPT-3 has a prompt limit of about ~2048 "tokens", which corresponds to about 4 characters in text. If my understanding is correct, a deep neural network is not learning after it is trained ...
0 votes
1 answer
256 views

Papers on Prompt Engineering

I am into AI in general and NLP in particular. Besides, I have a background in philosophy, and the new LLMs like GPT-3 seem to have exciting capabilities. I want to study prompt engineering (for ...
1 vote
0 answers
22 views

Which language model technique should be used with a low-resource language?

I am working with a language for which the amount of text available is relatively small (less than a billion words), what techniques exist? In particular, is it worth using a transformer or is it ...
0 votes
0 answers
54 views

BERT predictive distribution for non-masked tokens

In the BERT paper, the masking regime is described in the following way: The training data generator chooses 15% of the token positions at random for prediction. If the i-th token is chosen, we ...
  • 101
0 votes
2 answers
400 views

How to combine pretrained language models with additional feature set?

Are there any techniques to combine a feature set (other than the text itself) with pretrained language models. Let's say I have a random NLP task that tries to predict a binary class label based on e....
  • 101
0 votes
0 answers
13 views

What is the next-character perplexity of the PaLM model?

In the 2022 paper "PaLM: Scaling Language Modeling with Pathways", what is the bits-per-character perplexity of the resultant pre-trained model for next-word prediction?
  • 101
13 votes
1 answer
12k views

What is the "temperature" in the GPT models?

What does the temperature parameter mean when talking about the GPT models? I know that a higher temperature value means more randomness, but I want to know how randomness is introduced. Does ...
  • 253
0 votes
1 answer
347 views

Why do language models produce different outputs for same prompt?

For conventional 'Neural Networks', the weights simply act as a transformation in highly multi-dimensional space; for a forward pass, the output is always the same since there is no stochastic ...
  • 126
2 votes
1 answer
1k views

What is input (and shape) to K/V/Q of self-attention of EACH Decoder block of Language-translation model Transformer's tokens during Inference?

Transformer model of the original Attention paper has a decoder unit that works differently during Inference than Tranining. I'm trying to understand the shapes used during decoder (both self-...
  • 171
2 votes
1 answer
63 views

Are training sequences for LMs sampled in an IID fashion?

If I understand correctly, when training language models, we take a document and then chunk the document into a sequences of k tokens. So if the document is of length 30 and k=10, then we'll have 20 ...
  • 121
4 votes
2 answers
5k views

What is the difference between a language model and a word embedding?

I am self-studying applications of deep learning on the NLP and machine translation. I am confused about the concepts of "Language Model", "Word Embedding", "BLEU Score". ...
  • 153
0 votes
1 answer
425 views

Are there any good alternatives to an LSTM language model for text generation?

I have a trained LSTM language model and want to use it to generate text. The standard approach for this seems to be: Apply softmax function Take a weighted random choice to determine the next word ...
1 vote
0 answers
9 views

When we translate a text from one language to another, how does the frequency of various POS tags change?

When we translate a text from one language to another, how does the frequency of various POS tags change? So, let's say we have a text in English, with 10% nouns, 20% adjectives, 15% adverbs, 25% ...
1 vote
0 answers
143 views

How is input defined for a biaxial lstm network for generating music?

I am reading Composing Music With Recurrent Neural Networks by Daniel D. Johnson. But I am really confused about the input passed to this network. If we pass notes of music along the time axis, then ...
  • 121
1 vote
1 answer
974 views

Is there a way to provide multiple masks to BERT in MLM task?

I'm facing a situation where I've to fetch probabilities from BERT MLM for multiple words in a single sentence. ...
1 vote
1 answer
52 views

Appropriate metric and approach for natural language generation for small sentences

I am trying to create a language generation model to generate very short sentences/words, like a rapper name generator. The sentences in my dataset are anywhere between 1 word and 15 words (3-155 ...
3 votes
1 answer
151 views

Fundamentally, what is a perfect language model?

Suppose that we want to generate a sentence made of words according to language $L$: $$ W_1 W_2 \ldots W_n $$ Question: What is the perfect language model? I ask about perfect because I want to know ...
  • 151
1 vote
1 answer
483 views

What are the main differences between a language model and a machine translation model?

What are the main differences between a language model and a machine translation model?
  • 1,433
6 votes
1 answer
1k views

What are pros and cons of Bi-LSTM as compared to LSTM?

What are the pros and cons of LSTM vs Bi-LSTM in language modelling? What was the need to introduce Bi-LSTM?
  • 1,433
1 vote
0 answers
74 views

How to pad sequences during training for an encoder decoder model

I've got an encoder-decoder model for character level English language spelling correction, it is pretty basic stuff with a two LSTM encoder and another LSTM decoder. However, up until now, I have ...
7 votes
1 answer
1k views

How to use BERT as a multi-purpose conversational AI?

I'm looking to make an NLP model that can achieve a dual purpose. One purpose is that it can hold interesting conversations (conversational AI), and another being that it can do intent classification ...
  • 323
4 votes
3 answers
183 views

What would be a good internal language for an AI?

For an AI to represent the world, it would be good if it could translate human sentences into something more precise. We know, for example, that mathematics can be built up from set theory. So ...
  • 2,156
4 votes
2 answers
154 views

Are embeddings in multi-lingual language models comparable across languages?

Facebook has just pushed out a bigger version of their multi-lingual language model XLM, called XLM-R. My question is: do these kind of multi-lingual models imply, or even ensure, that their ...
2 votes
1 answer
63 views

How do you build a language model to predict the contextual similarity between two documents?

How do you build a language model to predict the contextual similarity between two documents?
2 votes
0 answers
65 views

How can I generate a document from a single word using GPT or BERT?

I have a dataset of 100000 documents each labelled with a topic to it. I want to create a model such that, given a topic, the model can generate a document from it. I came across language models GPT,...
5 votes
2 answers
2k views

Where can I find pre-trained language models in English and German? [closed]

Where can I find (more) pre-trained language models? I am especially interested in neural network-based models for English and German. I am aware only of Language Model on One Billion Word Benchmark ...