Questions tagged [language-model]

For questions related to the concept of a language model, which is a probability distribution over sequences of words (for example, of a natural language, such as English).

Filter by
Sorted by
Tagged with
2
votes
1answer
31 views

Are training sequences for LMs sampled in an IID fashion?

If I understand correctly, when training language models, we take a document and then chunk the document into a sequences of k tokens. So if the document is of length 30 and k=10, then we'll have 20 ...
1
vote
2answers
45 views

What is the difference between a language model and a word embedding?

I am self-studying applications of deep learning on the NLP and machine translation. I am confused about the concepts of "Language Model", "Word Embedding", "BLEU Score". ...
0
votes
0answers
10 views

Understanding Huggingface QA Transformers Confidence Scores

I seem to be unable to find information on whether the confidence scores for the huggingface transformers used for QA are in the form of 10^-2 or 10^-5. What are your thoughts on the matter? Example: <...
0
votes
1answer
68 views

Are there any good alternatives to an LSTM language model for text generation?

I have a trained LSTM language model and want to use it to generate text. The standard approach for this seems to be: Apply softmax function Take a weighted random choice to determine the next word ...
1
vote
0answers
4 views

POS Tag Frequency and Language Translation

When we translate a text from one language to another, how does the frequency of various POS tags change? So let's say we have a text in English with 10% nouns, 20% adjectives, 15% adverbs, 25% verbs, ...
1
vote
0answers
33 views

How is input defined for a biaxial lstm network for generating music?

I am reading Composing Music With Recurrent Neural Networks by Daniel D. Johnson. But I am really confused about the input passed to this network. If we pass notes of music along the time axis, then ...
1
vote
1answer
100 views

Is there a way to provide multiple masks to BERT in MLM task?

I'm facing a situation where I've to fetch probabilities from BERT MLM for multiple words in a single sentence. ...
1
vote
1answer
26 views

Appropriate metric and approach for natural language generation for small sentences

I am trying to create a language generation model to generate very short sentences/words, like a rapper name generator. The sentences in my dataset are anywhere between 1 word and 15 words (3-155 ...
3
votes
1answer
94 views

Fundamentally, what is a perfect language model?

Suppose that we want to generate a sentence made of words according to language $L$: $$ W_1 W_2 \ldots W_n $$ Question: What is the perfect language model? I ask about perfect because I want to know ...
0
votes
0answers
25 views

Comparing English and Spanish Queries using NLP

I am trying to see whether there are differences in sentences between the topics in 2 languages ie English and Spanish. Eg. (Face masks are mandatory, la mascarilla es obligatoria) but this stretched ...
0
votes
0answers
31 views

How can Kneser-Ney Smoothing be integrated into a neural language model?

I found a paper titled Multimodal representation: Kneser-Ney Smoothing/Skip-Gram based neural language model. I am curious about how can Kneser-Ney Smoothing technique be integrated into a feed-...
1
vote
1answer
36 views

What are the main differences between a language model and a machine translation model?

What are the main differences between a language model and a machine translation model?
1
vote
0answers
34 views

How to pad sequences during training for an encoder decoder model

I've got an encoder-decoder model for character level English language spelling correction, it is pretty basic stuff with a two LSTM encoder and another LSTM decoder. However, up until now, I have ...
6
votes
1answer
831 views

How to use BERT as a multi-purpose conversational AI?

I'm looking to make an NLP model that can achieve a dual purpose. One purpose is that it can hold interesting conversations (conversational AI), and another being that it can do intent classification ...
4
votes
3answers
146 views

What would be a good internal language for an AI?

For an AI to represent the world, it would be good if it could translate human sentences into something more precise. We know, for example, that mathematics can be built up from set theory. So ...
4
votes
2answers
91 views

Are embeddings in multi-lingual language models comparable across languages?

Facebook has just pushed out a bigger version of their multi-lingual language model XLM, called XLM-R. My question is: do these kind of multi-lingual models imply, or even ensure, that their ...
2
votes
1answer
51 views

How do you build a language model to predict the contextual similarity between two documents?

How do you build a language model to predict the contextual similarity between two documents?
2
votes
0answers
42 views

How can I generate a document from a single word using GPT or BERT?

I have a dataset of 100000 documents each labelled with a topic to it. I want to create a model such that, given a topic, the model can generate a document from it. I came across language models GPT,...
4
votes
2answers
1k views

Where can I find pre-trained language models in English and German?

Where can I find (more) pre-trained language models? I am especially interested in neural network-based models for English and German. I am aware only of Language Model on One Billion Word Benchmark ...