Questions tagged [natural-language-processing]

For questions related to natural language processing (NLP), which is concerned with the interactions between computers and human (or natural) languages, in particular how to create programs that process and analyze large amounts of natural language data.

228 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
8 votes
1 answer
179 views

Why are documents kept separated when training a text classifier?

Most of the literature considers text classification as the classification of documents. When using the bag-of-words and Bayesian classification, they usually use the statistic TF-IDF, where TF ...
user avatar
  • 236
6 votes
1 answer
139 views

Are there transformer-based architectures that can produce fixed-length vector encodings given arbitrary-length text documents?

BERT encodes a piece of text such that each token (usually words) in the input text map to a vector in the encoding of the text. However, this makes the length of the encoding vary as a function of ...
user avatar
4 votes
0 answers
32 views

Why does all of NLP literature use noise contrastive estimation loss for negative sampling instead of sampled softmax loss?

A sampled softmax function is like a regular softmax but randomly selects a given number of 'negative' samples. This is difference than NCE Loss, which doesn't use a softmax at all, it uses a ...
user avatar
4 votes
0 answers
55 views

What is the motivation for row-wise convolution and folding in Kalchbrenner et al. (2014)?

I was reading the paper by Kalchbrenner et al. titled A Convolutional Neural Network for Modelling Sentences and I am struggling to understand their definition of convolutional layer. First, let's ...
user avatar
4 votes
0 answers
421 views

RL to generate sentences

I want to develop a system to generate grammatically correct sentences. The input would be some words. The output would be a grammatically correct human-like sentence. Eg: Input: capital, Paris, ...
user avatar
  • 161
3 votes
0 answers
58 views

What is the difference between zero-padding and character-padding in Recurrent Neural Networks?

For RNN's to work efficiently, we vectorize the operations, which results in an input matrix of shape (m, max_seq_len) where m ...
user avatar
3 votes
1 answer
47 views

Is there a complement to GPT/2/3 that can be trained using supervised learning methods?

This is a bit of a soft question, not sure if it's on topic, please let me know how I can improve it if it doesn't meet the criteria for the site. GPT models are unsupervised in nature and are (from ...
user avatar
3 votes
1 answer
326 views

What are the keys and values of the attention model for the encoder and decoder in the "Attention Is All You Need" paper?

I have recently encountered the paper on NLP. It is very new to me and I am still unable to see how that works. I have used all the resources over there from the original paper to Youtube videos and ...
user avatar
  • 233
3 votes
0 answers
86 views

Is there a good book or paper on word embeddings?

Is there a good and modern book that focuses on word embeddings and their applications? It would also be ok to provide the name of a paper that provides a good overview of word embeddings.
user avatar
  • 889
3 votes
1 answer
72 views

What is meant by the expected BLEU cost when training with BLEU and SIMILE?

Recently I was reading a paper based on a new evaluation metric SIMILE. In a section, validation loss comparison had been made for SIMILE and BLEU. The plot showed the expected BLEU cost when training ...
user avatar
3 votes
0 answers
55 views

Could zero-padding affect learning in a negative way?

I implemented an LSTM with Keras to perform word ordering task (given a syntactically unordered sentence, the goal is to label ...
user avatar
  • 133
3 votes
0 answers
392 views

Can Bert be used to extract embedding for large categorical features?

I've lot of training data points (i.e in millions) and I've around few features but the issue with that is all the features are categorical data with 1 million+ categories in each. So, I couldn't use ...
user avatar
  • 149
3 votes
0 answers
83 views

Creating an AI than can learn to give instructions

So we think a computer is dumb because it can only follow instructions. Therefor I am trying to create an AI that can give instructions. The idea is this: Create a geometric scene (A) then make a ...
user avatar
  • 2,054
3 votes
0 answers
36 views

Why embedding layer is used in the character-level Natural Language Processing models

Problem Background I am working with a problem, which requires a character-level, deep learning model. Previously I was working with word-level deep NLP (Natural Language Processing) models, and in ...
user avatar
3 votes
0 answers
254 views

How to use TPU for real-time low-latency inference?

I use Google's Cloud TPU hardware extensively using Tensorflow for training models and inference, however, when I run inference I do it in large batches. The TPU takes about 3 minutes to warm up ...
user avatar
  • 51
3 votes
0 answers
27 views

Can computers recognise "grouping" from voice tonality?

In human communication, tonality or tonal language play many complex information, including emotions and motives. But excluding such complex aspects, tonality serves some a very basic purpose of "...
user avatar
3 votes
0 answers
205 views

How does bidirectional encoding allow the predicted word to indirectly "see itself"?

Before the release of BERT, we used to say that it is not possible to train bidirectional models by simply conditioning each word on its previous and next words, since this would allow the word that's ...
user avatar
  • 409
3 votes
0 answers
39 views

How would an AI visualize a story written in natural language?

Can AI transform natural language text describing real scenarios to visual images and videos ? How does as AI interprets say a Harry Potter story if it has to reproduce it in form of videos ? Would be ...
user avatar
  • 41
3 votes
2 answers
126 views

What is the difference between automatic transcription and automatic speech recognition?

What is the difference between automatic transcription and automatic speech recognition? Are they the same? Is my following interpretation correct? Automatic transcription: it converts the speech to ...
user avatar
  • 141
3 votes
0 answers
39 views

Extracting referenced documents

I'm looking to write an AI that will be able to extract in text references from standards documents to assist human research. My use case is extracting the identifying numbers, for example, "AR 25-2",...
user avatar
3 votes
0 answers
42 views

What is the "question" when using Dynamic Memory Networks to do part-of-speech tagging and sentiment analysis?

In the paper Ask Me Anything: Dynamic Memory Networks for Natural Language Processing the authors described a Dynamic Memory Network in the context of question answering. Then, they also tested the ...
user avatar
  • 139
3 votes
0 answers
211 views

Has anyone used YodaQA for natural language processing?

Has anyone used YodaQA for natural language processing? How easy is it to link to a document database other than Wikipedia? We're thinking we can create a bot to use AI to analyze our developer and ...
user avatar
2 votes
0 answers
37 views

Why use a fully connected layer for attention?

In the paper Neural Machine Translation by Jointly Learning to Align and Translate, attention is used with a single fully connected layer. Specifically, in the auto-regressive set up (equation 4), the ...
user avatar
  • 121
2 votes
0 answers
21 views

Embedding from Transformer-based model from paragraph or documnet (like Doc2Vec)

I have a set of data that contains the different lengths of sequences. On average the sequence length is 600. The dataset is like this: ...
user avatar
2 votes
0 answers
60 views

How to Select Model Parameters for Transformer (Heads, number of layers, etc)

Is there a general guideline on how the Transformer model parameters should be selected, or the range of these parameters that should be included in a hyperparameter sweep? Number of heads Number of ...
user avatar
2 votes
0 answers
67 views

Extracting "hidden" costs from financial statements using NLP

I'm designing a NLP model to extract various kinds of "hidden" expenses from 10-K and 10-Q financial statements. I've come up with about 7 different expense categories (restructuring costs, ...
user avatar
  • 21
2 votes
0 answers
24 views

Adding corpus to BERT for QA

I was wondering about SciBERT's QA abilities using SQuAD. I have a scarce textual dataset consisting of less than 100 files where doctors are discussing cancer in dialogues. I want to add it to ...
user avatar
2 votes
0 answers
191 views

T5 or BERT for sentence correction/generation task?

I have sentences with some grammatical errors , with no punctuations and digits written in words... something like below: As you can observe, a proper noun , winston isnt highlighted with capital in ...
user avatar
2 votes
0 answers
170 views

How to handle long sequences with transformers?

I have a time series sequence with 10 million steps. In step $t$, I have a 400 dimensional feature vector $X_t$ and a scalar value $y_t$ which I want to predict during inference time and I know during ...
user avatar
  • 83
2 votes
0 answers
40 views

Can One-Hot Vectors be used as Inputs for Recurrent Neural Networks?

When using an RNN to encode a sentence, one normally takes each word, passes it through an embedding layer, and then uses the dense embedding as the input into the RNN. Lets say instead of using dense ...
user avatar
2 votes
0 answers
53 views

What is the difference between text-based image retrieval and natural language object retrieval?

I'm working on creating a model that locates the object in the scene (2D image or 3D scene) using a natural language query. I came across this paper on natural language object retrieval, which ...
user avatar
  • 21
2 votes
0 answers
805 views

Why aren't the BERT layers frozen during fine-tuning tasks?

During transfer learning in computer vision, I've seen that the layers of the base model are frozen if the images aren't too different from the model on which the base model is trained on. However, on ...
user avatar
2 votes
0 answers
51 views

Estimating an $n$-Gram model using on bigrams

One of the main arguments against $n$-gram models is that, as $n$ increases, there is no way to compute $P(w_n|w_1,\cdots,w_{n-1})$ from training data (since the chance of visiting $w_n,...,w_1$ is ...
user avatar
  • 121
2 votes
0 answers
80 views

NLP Bible verse division problem: Whats the best model/method?

I'm working on a project compiling various versions of the Bible into a dataset. For the most part versions separate verses discreetly. In some versions, however, verses are combined. Instead of verse ...
user avatar
  • 21
2 votes
0 answers
41 views

How should I design a reward function for a NLP problem where two models interoperate?

I would like to design a reward function. I am training two models from the first model that classify set of texts (paragraphs and keywords) and I also got some hidden states. The second model is ...
user avatar
  • 21
2 votes
1 answer
129 views

How do LSTM and GRU avoid to overcome the vanishing gradient problem?

I'm watching the video Recurrent Neural Networks (RNN) | RNN LSTM | Deep Learning Tutorial | Tensorflow Tutorial | Edureka where the author says that the LSTM and GRU architecture help to reduce the ...
user avatar
  • 1,243
2 votes
0 answers
29 views

How does one detect linguistic recursion so as to know how much nesting there is, if any?

To be clear, recursion in linguistics is here better called "nesting" in this CS context to avoid confusing it with the other recursion. How does one detect nesting? I am particularly ...
user avatar
2 votes
0 answers
148 views

What are the advantages and disadvantages of extrinsic and perplexity model evaluation in NLP?

In the video Evaluation and Perplexity by Dan Jurafsky, the author talks about extrinsic and perplexity evaluation in the context of natural language processing (NLP). What are the advantages and ...
user avatar
  • 1,243
2 votes
0 answers
61 views

How does positional encoding work in the transformer model?

In the transformer model, to incorporate positional information of texts, the researchers have added a positional encoding to the model. How does positional encoding work? How does the positional ...
user avatar
  • 946
2 votes
0 answers
20 views

Sign Language to Speech conversion

Is there any solution about sign language to speech conversion for mobiles? Can anyone suggest me the flow and tools so that I may implement the solution for mobiles?
user avatar
2 votes
0 answers
222 views

Can we use GPT-2 to smooth out / correct text?

Are we able to use models like GPT-2 to smooth out/correct text? For instance if I have two paragraphs that need some text to make the transition easier to read, could this text be generated? And, ...
user avatar
  • 121
2 votes
0 answers
23 views

How to tell if two hotel reviews addressing the same thing

I am playing with a large dataset of hotel reviews, which contains both positive and negative reviews (the reviews are labeled). I want to use this dataset to perform textual style transfer - given a ...
user avatar
2 votes
0 answers
38 views

How can I extract the reason of the legal compensation from a court report?

I'm working on a project (court-related). At a certain point, I have to extract the reason of the legal compensation. For instance, let's take these sentences (from a court report) Order mister X ...
user avatar
2 votes
0 answers
94 views

Is NLP likely to be sufficiently solved in the next few years?

The reason I am asking this question is because I am about to start a PhD in NLP. So I am wondering if there would be as much job opportunities in research in industry as oppose to in academia in the ...
user avatar
  • 21
2 votes
0 answers
45 views

What role do distractors play in natural language processing?

I’m doing research on natural language processing (NLP). I’d like to put together my own model. However, I'm running into a concept I am not familiar with, namely, distractors. A google search does ...
user avatar
  • 323
2 votes
0 answers
42 views

Are there any approaches other than deep learning to deal with unexpected questions in a question answering system?

I'm working on a question answering bot as my graduation project. The main concept is having a text file with many sentences, and building a question answering bot which answers a user's question ...
user avatar
2 votes
0 answers
61 views

How can I feed any word into a neural network?

I am working on an Intent detection problem for a chatbot in Java. So I need to convert words from String to a double[] format. I tried using wordToVec(deeplearning4j), but it does not return a vector ...
user avatar
2 votes
0 answers
28 views

NLP annotation tool online and other tools to compare performances of different NLP algorithms

I do text annotations (POS tagging, NER, chunking, synset) by using a specific annotation tool for Natural Language Processing. I would like to make the same annotations on different tools to compare ...
user avatar
  • 163
2 votes
0 answers
30 views

Is there a detailed description or implementation of an end-to-end speech recognition system?

I am currently trying to implement an end-to-end speech recognition system from scratch, that is, without using any of the existing frameworks (like TensorFlow, Keras, etc.). I am building my own ...
user avatar
  • 121
2 votes
0 answers
375 views

How to Extract Information from the Image

I'm trying to extract some particular information from the image(png). I tried to extract the text using the below code ...
user avatar
  • 1,193

1
2 3 4 5