Questions tagged [natural-language-processing]

For questions related to natural language processing (NLP), which is concerned with the interactions between computers and human (or natural) languages, in particular how to create programs that process and analyze large amounts of natural language data.

279 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
8 votes
1 answer
813 views

Are there transformer-based architectures that can produce fixed-length vector encodings given arbitrary-length text documents?

BERT encodes a piece of text such that each token (usually words) in the input text map to a vector in the encoding of the text. However, this makes the length of the encoding vary as a function of ...
HelloGoodbye's user avatar
6 votes
2 answers
484 views

How does GPT-based language model like ChatGPT determine the n-th letter of a word?

I understand that GPT models process input text by converting words into tokens and then embedding vectors and do not process them letter by letter. Given this approach, I am curious to know how a ...
Peyman's user avatar
  • 564
5 votes
0 answers
603 views

Are "prompt engineering" and "prompt design" used as synonymous?

Are "prompt engineering" and "prompt design" used as synonymous / equivalent terms on the day to day communications (not research papers) in Artificial Intelligence community ? Do ...
Rubén's user avatar
  • 159
4 votes
1 answer
2k views

Why aren't the BERT layers frozen during fine-tuning tasks?

During transfer learning in computer vision, I've seen that the layers of the base model are frozen if the images aren't too different from the model on which the base model is trained on. However, on ...
Bunny Rabbit's user avatar
4 votes
0 answers
127 views

Could zero-padding affect learning in a negative way?

I implemented an LSTM with Keras to perform word ordering task (given a syntactically unordered sentence, the goal is to label ...
pairon's user avatar
  • 143
4 votes
0 answers
34 views

Why does all of NLP literature use noise contrastive estimation loss for negative sampling instead of sampled softmax loss?

A sampled softmax function is like a regular softmax but randomly selects a given number of 'negative' samples. This is difference than NCE Loss, which doesn't use a softmax at all, it uses a ...
SantoshGupta7's user avatar
4 votes
0 answers
60 views

What is the motivation for row-wise convolution and folding in Kalchbrenner et al. (2014)?

I was reading the paper by Kalchbrenner et al. titled A Convolutional Neural Network for Modelling Sentences and I am struggling to understand their definition of convolutional layer. First, let's ...
Tomasz Garbus's user avatar
4 votes
0 answers
440 views

RL to generate sentences

I want to develop a system to generate grammatically correct sentences. The input would be some words. The output would be a grammatically correct human-like sentence. Eg: Input: capital, Paris, ...
MD Luffy's user avatar
  • 161
3 votes
0 answers
334 views

How to work with multiple embeddings?

This is a conceptual gap that I have concerning embeddings, and would really appreciate some help closing it. I understand when you embed a corpus for, let's say, a question-and-answer task you can ...
Ian Murray's user avatar
3 votes
0 answers
233 views

Let's Verify Step by Step: Old wine in new bottles?

In their paper "Let's Verify Step by Step" OpenAI proudly presents a new way of reward learning which shall foster LLMs' capabilities of mathematical and logical reasoning: We've trained a ...
Hans-Peter Stricker's user avatar
3 votes
0 answers
319 views

How is ChatGPT able to perform part-of-speech tagging?

ChatGPT seems to be able of part-of-speech tagging: How can its – possibly emergent – ability to perform part-of-speech tagging be understood?
Hans-Peter Stricker's user avatar
3 votes
0 answers
510 views

How to Select Model Parameters for Transformer (Heads, number of layers, etc)

Is there a general guideline on how the Transformer model parameters should be selected, or the range of these parameters that should be included in a hyperparameter sweep? Number of heads Number of ...
Athena Wisdom's user avatar
3 votes
0 answers
97 views

What is the difference between zero-padding and character-padding in Recurrent Neural Networks?

For RNN's to work efficiently, we vectorize the operations, which results in an input matrix of shape (m, max_seq_len) where m ...
PhysicsMan's user avatar
3 votes
0 answers
449 views

T5 or BERT for sentence correction/generation task?

I have sentences with some grammatical errors , with no punctuations and digits written in words... something like below: As you can observe, a proper noun , winston isnt highlighted with capital in ...
Varun kadekar's user avatar
3 votes
1 answer
849 views

What are the keys and values of the attention model for the encoder and decoder in the "Attention Is All You Need" paper?

I have recently encountered the paper on NLP. It is very new to me and I am still unable to see how that works. I have used all the resources over there from the original paper to Youtube videos and ...
Deshwal's user avatar
  • 253
3 votes
0 answers
113 views

Is there a good book or paper on word embeddings?

Is there a good and modern book that focuses on word embeddings and their applications? It would also be ok to provide the name of a paper that provides a good overview of word embeddings.
ddaedalus's user avatar
  • 919
3 votes
0 answers
551 views

Can Bert be used to extract embedding for large categorical features?

I've lot of training data points (i.e in millions) and I've around few features but the issue with that is all the features are categorical data with 1 million+ categories in each. So, I couldn't use ...
user_12's user avatar
  • 149
3 votes
0 answers
178 views

Creating an AI than can learn to give instructions

So we think a computer is dumb because it can only follow instructions. Therefor I am trying to create an AI that can give instructions. The idea is this: Create a geometric scene (A) then make a ...
zooby's user avatar
  • 2,196
3 votes
0 answers
39 views

Why embedding layer is used in the character-level Natural Language Processing models

Problem Background I am working with a problem, which requires a character-level, deep learning model. Previously I was working with word-level deep NLP (Natural Language Processing) models, and in ...
Daniel Wiczew's user avatar
3 votes
0 answers
443 views

How to use TPU for real-time low-latency inference?

I use Google's Cloud TPU hardware extensively using Tensorflow for training models and inference, however, when I run inference I do it in large batches. The TPU takes about 3 minutes to warm up ...
adng's user avatar
  • 51
3 votes
0 answers
29 views

Can computers recognise "grouping" from voice tonality?

In human communication, tonality or tonal language play many complex information, including emotions and motives. But excluding such complex aspects, tonality serves some a very basic purpose of "...
Always Confused's user avatar
3 votes
0 answers
265 views

How does bidirectional encoding allow the predicted word to indirectly "see itself"?

Before the release of BERT, we used to say that it is not possible to train bidirectional models by simply conditioning each word on its previous and next words, since this would allow the word that's ...
HLeb's user avatar
  • 579
3 votes
0 answers
44 views

How would an AI visualize a story written in natural language?

Can AI transform natural language text describing real scenarios to visual images and videos ? How does as AI interprets say a Harry Potter story if it has to reproduce it in form of videos ? Would be ...
katipra's user avatar
  • 41
3 votes
0 answers
43 views

Extracting referenced documents

I'm looking to write an AI that will be able to extract in text references from standards documents to assist human research. My use case is extracting the identifying numbers, for example, "AR 25-2",...
comp.sci.intern's user avatar
3 votes
0 answers
114 views

NLP proved against US legal texts

I'm new to AI development and am looking for a quality algorithm (potentially nlp?) implementation proved against US legal texts. Obviously some training would need to be done, but I've found little ...
Chris Giddings's user avatar
3 votes
0 answers
43 views

What is the "question" when using Dynamic Memory Networks to do part-of-speech tagging and sentiment analysis?

In the paper Ask Me Anything: Dynamic Memory Networks for Natural Language Processing the authors described a Dynamic Memory Network in the context of question answering. Then, they also tested the ...
mauna's user avatar
  • 139
3 votes
0 answers
215 views

Has anyone used YodaQA for natural language processing?

Has anyone used YodaQA for natural language processing? How easy is it to link to a document database other than Wikipedia? We're thinking we can create a bot to use AI to analyze our developer and ...
Charles Miller's user avatar
2 votes
0 answers
32 views

How to learn text style in an article using LLMs?

What is the best way to learn text style in an article? By text style I mean special characteristics and patterns inherent to different authors/group's writing style. For-example, author attribution ...
Shayan's user avatar
  • 21
2 votes
1 answer
38 views

Why is the sinusoidal model classified as absolute positional encoding in some literature?

I am currently reading in depth about positional encodings, and as we know there are two types of positional encodings: Absolute and relative. My question: Why is the sinusoidal model classified as ...
Ali Haider Ahmad's user avatar
2 votes
0 answers
84 views

What is the meaning of "dimensionality of the embeddings and hidden states"?

I was reading the GPT-2 and LSTM documents and noticed that they use the terms "dimension of embedding and hidden state". For GPT-2, the size is $768$, and for LSTM, the size is $256$. What ...
user avatar
2 votes
0 answers
65 views

Why is an encoder + decoder model with L by L layers the same speed as as decoder only model with 2 L layers?

I was watching this lecture: https://youtu.be/27rNqGrTdSI?t=2295 In it the presenter stated that: "An encoder + decoder model with L by L layers is actually the same speed as as decoder only ...
shawn's user avatar
  • 21
2 votes
0 answers
2k views

Any models for text to json

There are many sequence to sequence (seq2seq) models and end to end models, like text to sql. I was wondering are there any text to json deep learning models? For example: Text ...
tired and bored dev's user avatar
2 votes
1 answer
1k views

If GPT-3 is trained on predicting the next token, how is it able to take commands?

From my understanding, GPT-3 is trained on predicting the next token from a sequence of tokens. Given this, how is it able to take commands? For instance, in this example input, wouldn't the ...
Andrew Tang's user avatar
2 votes
0 answers
357 views

Why use a fully connected layer for attention?

In the paper Neural Machine Translation by Jointly Learning to Align and Translate, attention is used with a single fully connected layer. Specifically, in the auto-regressive set up (equation 4), the ...
Vishaal's user avatar
  • 121
2 votes
1 answer
145 views

How would the probability of a document $P(d)$ be computed in the Naive Bayes classifier?

In naive Bayes classification, we estimate the class of a document as follows $$\hat{c} = \arg \max_{c \in C} P(c \mid d) = \arg \max_{c \in C} \dfrac{ P(d \mid c)P(c) }{P(d)} $$ It has been said in ...
hanugm's user avatar
  • 3,792
2 votes
0 answers
26 views

Embedding from Transformer-based model from paragraph or documnet (like Doc2Vec)

I have a set of data that contains the different lengths of sequences. On average the sequence length is 600. The dataset is like this: ...
Bloodstone Programmer's user avatar
2 votes
0 answers
83 views

Extracting "hidden" costs from financial statements using NLP

I'm designing a NLP model to extract various kinds of "hidden" expenses from 10-K and 10-Q financial statements. I've come up with about 7 different expense categories (restructuring costs, ...
hohner's user avatar
  • 21
2 votes
0 answers
38 views

Adding corpus to BERT for QA

I was wondering about SciBERT's QA abilities using SQuAD. I have a scarce textual dataset consisting of less than 100 files where doctors are discussing cancer in dialogues. I want to add it to ...
DarknessPlusPlus's user avatar
2 votes
0 answers
308 views

How to handle long sequences with transformers?

I have a time series sequence with 10 million steps. In step $t$, I have a 400 dimensional feature vector $X_t$ and a scalar value $y_t$ which I want to predict during inference time and I know during ...
mhsnk's user avatar
  • 123
2 votes
0 answers
98 views

Can One-Hot Vectors be used as Inputs for Recurrent Neural Networks?

When using an RNN to encode a sentence, one normally takes each word, passes it through an embedding layer, and then uses the dense embedding as the input into the RNN. Lets say instead of using dense ...
chessprogrammer's user avatar
2 votes
0 answers
89 views

What is the difference between text-based image retrieval and natural language object retrieval?

I'm working on creating a model that locates the object in the scene (2D image or 3D scene) using a natural language query. I came across this paper on natural language object retrieval, which ...
Sid's user avatar
  • 21
2 votes
0 answers
258 views

How are weight matrices in attention learned?

I have been looking into transformers lately and have been reading tons of tutorials. All of them address the intuition behind attention, which I understand. However, they treat learning the weight ...
Ege Demir's user avatar
2 votes
0 answers
54 views

Estimating an $n$-Gram model using on bigrams

One of the main arguments against $n$-gram models is that, as $n$ increases, there is no way to compute $P(w_n|w_1,\cdots,w_{n-1})$ from training data (since the chance of visiting $w_n,...,w_1$ is ...
Amir's user avatar
  • 121
2 votes
0 answers
131 views

NLP Bible verse division problem: Whats the best model/method?

I'm working on a project compiling various versions of the Bible into a dataset. For the most part versions separate verses discreetly. In some versions, however, verses are combined. Instead of verse ...
rwreed's user avatar
  • 121
2 votes
0 answers
79 views

How should I design a reward function for a NLP problem where two models interoperate?

I would like to design a reward function. I am training two models from the first model that classify set of texts (paragraphs and keywords) and I also got some hidden states. The second model is ...
No Na's user avatar
  • 21
2 votes
2 answers
337 views

How do LSTM and GRU avoid to overcome the vanishing gradient problem?

I'm watching the video Recurrent Neural Networks (RNN) | RNN LSTM | Deep Learning Tutorial | Tensorflow Tutorial | Edureka where the author says that the LSTM and GRU architecture help to reduce the ...
DRV's user avatar
  • 1,673
2 votes
0 answers
32 views

How does one detect linguistic recursion so as to know how much nesting there is, if any?

To be clear, recursion in linguistics is here better called "nesting" in this CS context to avoid confusing it with the other recursion. How does one detect nesting? I am particularly ...
JohnnyApplesauce's user avatar
2 votes
0 answers
246 views

What are the advantages and disadvantages of extrinsic and perplexity model evaluation in NLP?

In the video Evaluation and Perplexity by Dan Jurafsky, the author talks about extrinsic and perplexity evaluation in the context of natural language processing (NLP). What are the advantages and ...
DRV's user avatar
  • 1,673
2 votes
0 answers
161 views

How does positional encoding work in the transformer model?

In the transformer model, to incorporate positional information of texts, the researchers have added a positional encoding to the model. How does positional encoding work? How does the positional ...
Eka's user avatar
  • 1,066
2 votes
0 answers
20 views

Sign Language to Speech conversion

Is there any solution about sign language to speech conversion for mobiles? Can anyone suggest me the flow and tools so that I may implement the solution for mobiles?
sreehari's user avatar

1
2 3 4 5 6