Questions tagged [language-model]

For questions related to the concept of a language model, which is a probability distribution over sequences of words (for example, of a natural language, such as English).

Filter by
Sorted by
Tagged with
0 votes
0 answers
12 views

What is the next-character perplexity of the PaLM model?

In the 2022 paper "PaLM: Scaling Language Modeling with Pathways", what is the bits-per-character perplexity of the resultant pre-trained model for next-word prediction?
user avatar
  • 101
0 votes
0 answers
30 views

Modify the architecture of the pre-trained model

I'm trying to remove some layers and reduce the size of fully connected in pre-trained BERT model. And then I'm going to fine-tune the modified models on GLUE. But the problem is that the modified ...
user avatar
  • 1
0 votes
0 answers
21 views

How does Stack-Augmented Recurrent Nets in work?

I am new to RNN/LSTM and I am working on a project about language modeling. I just got familiarized with simple RNN and LSTM. However, these simple models did not achieve the performance I want. Since ...
user avatar
4 votes
1 answer
1k views

What is the "temperature" in the GPT models?

What does the temperature parameter mean when talking about the GPT models? I know that a higher temperature value means more randomness, but I want to know how randomness is introduced. Does ...
user avatar
  • 163
0 votes
1 answer
59 views

Why do language models produce different outputs for same prompt?

For conventional 'Neural Networks', the weights simply act as a transformation in highly multi-dimensional space; for a forward pass, the output is always the same since there is no stochastic ...
user avatar
  • 126
0 votes
1 answer
312 views

What is input (and shape) to K/V/Q of self-attention of EACH Decoder block of Language-translation model Transformer's tokens during Inference?

Transformer model of the original Attention paper has a decoder unit that works differently during Inference than Tranining. I'm trying to understand the shapes used during decoder (both self-...
user avatar
  • 151
2 votes
1 answer
40 views

Are training sequences for LMs sampled in an IID fashion?

If I understand correctly, when training language models, we take a document and then chunk the document into a sequences of k tokens. So if the document is of length 30 and k=10, then we'll have 20 ...
user avatar
  • 121
4 votes
2 answers
2k views

What is the difference between a language model and a word embedding?

I am self-studying applications of deep learning on the NLP and machine translation. I am confused about the concepts of "Language Model", "Word Embedding", "BLEU Score". ...
user avatar
0 votes
1 answer
272 views

Are there any good alternatives to an LSTM language model for text generation?

I have a trained LSTM language model and want to use it to generate text. The standard approach for this seems to be: Apply softmax function Take a weighted random choice to determine the next word ...
user avatar
1 vote
0 answers
8 views

When we translate a text from one language to another, how does the frequency of various POS tags change?

When we translate a text from one language to another, how does the frequency of various POS tags change? So, let's say we have a text in English, with 10% nouns, 20% adjectives, 15% adverbs, 25% ...
user avatar
1 vote
0 answers
114 views

How is input defined for a biaxial lstm network for generating music?

I am reading Composing Music With Recurrent Neural Networks by Daniel D. Johnson. But I am really confused about the input passed to this network. If we pass notes of music along the time axis, then ...
user avatar
1 vote
1 answer
617 views

Is there a way to provide multiple masks to BERT in MLM task?

I'm facing a situation where I've to fetch probabilities from BERT MLM for multiple words in a single sentence. ...
user avatar
1 vote
1 answer
47 views

Appropriate metric and approach for natural language generation for small sentences

I am trying to create a language generation model to generate very short sentences/words, like a rapper name generator. The sentences in my dataset are anywhere between 1 word and 15 words (3-155 ...
user avatar
3 votes
1 answer
102 views

Fundamentally, what is a perfect language model?

Suppose that we want to generate a sentence made of words according to language $L$: $$ W_1 W_2 \ldots W_n $$ Question: What is the perfect language model? I ask about perfect because I want to know ...
user avatar
  • 151
1 vote
1 answer
228 views

What are the main differences between a language model and a machine translation model?

What are the main differences between a language model and a machine translation model?
user avatar
  • 1,243
4 votes
1 answer
527 views

What are pros and cons of Bi-LSTM as compared to LSTM?

What are the pros and cons of LSTM vs Bi-LSTM in language modelling? What was the need to introduce Bi-LSTM?
user avatar
  • 1,243
1 vote
0 answers
55 views

How to pad sequences during training for an encoder decoder model

I've got an encoder-decoder model for character level English language spelling correction, it is pretty basic stuff with a two LSTM encoder and another LSTM decoder. However, up until now, I have ...
user avatar
7 votes
1 answer
1k views

How to use BERT as a multi-purpose conversational AI?

I'm looking to make an NLP model that can achieve a dual purpose. One purpose is that it can hold interesting conversations (conversational AI), and another being that it can do intent classification ...
user avatar
  • 323
4 votes
3 answers
172 views

What would be a good internal language for an AI?

For an AI to represent the world, it would be good if it could translate human sentences into something more precise. We know, for example, that mathematics can be built up from set theory. So ...
user avatar
  • 2,054
4 votes
2 answers
110 views

Are embeddings in multi-lingual language models comparable across languages?

Facebook has just pushed out a bigger version of their multi-lingual language model XLM, called XLM-R. My question is: do these kind of multi-lingual models imply, or even ensure, that their ...
user avatar
2 votes
1 answer
58 views

How do you build a language model to predict the contextual similarity between two documents?

How do you build a language model to predict the contextual similarity between two documents?
user avatar
2 votes
0 answers
58 views

How can I generate a document from a single word using GPT or BERT?

I have a dataset of 100000 documents each labelled with a topic to it. I want to create a model such that, given a topic, the model can generate a document from it. I came across language models GPT,...
user avatar
5 votes
2 answers
2k views

Where can I find pre-trained language models in English and German? [closed]

Where can I find (more) pre-trained language models? I am especially interested in neural network-based models for English and German. I am aware only of Language Model on One Billion Word Benchmark ...
user avatar