Questions tagged [natural-language-processing]

For questions related to natural language processing (NLP), which is concerned with the interactions between computers and human (or natural) languages, in particular how to create programs that process and analyze large amounts of natural language data.

Filter by
Sorted by
Tagged with
1
vote
1answer
159 views

Can we use a pre trained Encoder (BERT, XLM ) with a Decoder (GPT, Transformer-XL) to build a Chatbot instead of Language Translation?

I was wondering if the BART or T5 models can do the task of generating sentences in English. Most of the models I have mentioned ...
1
vote
1answer
77 views

When to use NLP, NLG and NLU in conversation agents?

I had read some blogs (like 1, 2 or 3) about what the difference between all three of them is. I am trying to build an open domain conversation agent using natural language AI. That agent can do ...
0
votes
0answers
23 views

How to find difference between topics in two languages using NLP

I want to analyze queries and their differences between two different languages English and Spanish in this case. I'm aware of topic modeling. I'm in search of any corpus available or any algorithm to ...
0
votes
0answers
40 views

Which NLP Parser for Morse conversations?

In ham radio, Morse code is used in CW transmissions (yes even today). My project is to take already decoded Morse code text strings from an existing decoder and then recognize different ...
3
votes
1answer
160 views

What are the keys and values of the attention model for the encoder and decoder in the “Attention Is All You Need” paper?

I have recently encountered the paper on NLP. It is very new to me and I am still unable to see how that works. I have used all the resources over there from the original paper to Youtube videos and ...
0
votes
1answer
60 views

Why does the BERT NSP head linear layer have two outputs?

Here's the code in question. https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_bert.py#L491 ...
1
vote
0answers
28 views

Low accuracy during training for text summarization

I am trying to implement an extractive text summarization model. I am using keras and tensorflow. I have used bert sentence embeddings and the embeddings are fed into an LSTM layer and then to a Dense ...
1
vote
0answers
30 views

Two questions about the architecture of Google Bert model (in particular about parameters)

I'm looking for someone who can help me clarify a few details regarding the architecture of Bert model. Those details are necessary for me to come with a full understanding of Bert model, so your help ...
4
votes
1answer
405 views

What is the intuition behind the attention mechanism?

Attention idea is one of the most influential ideas in deep learning. The main idea behind attention technique is that it allows the decoder to "look back” at the complete input and extracts ...
1
vote
0answers
62 views

Transformer encoding for regression

I have a string of characters encoding a molecule. I want to regress some properties of those molecules. I tried using an LSTM that encodes all one hot encdoed characters, and then I take the last ...
1
vote
2answers
301 views

Why does this multiplication of $Q$ and $K$ have a variance of $d_k$, in scaled dot product attention?

In scaled dot product attention, we scale our outputs by dividing the dot product by the square root of the dimensionality of the matrix: The reason why is stated that this constrains the ...
3
votes
1answer
49 views

How can I make ELIZA more realistic?

I’ve coded a simple ELIZA chatbot for a high school coding competition. The chatbot is part of an app that’s designed to help its user cope with depression, anxiety, and similar mental health ...
4
votes
1answer
92 views

How is the Jacobian a generalisation of the gradient?

I came across these slides Natural Language Processing with Deep Learning CS224N/Ling284, in the context of natural language processing, which talk about the Jacobian as a generalization of the ...
1
vote
0answers
47 views

Sentence Segmentation for “Bullets & Numbering”

I am trying to input text into my word processor to be split into sentences first and then into words. An example paragraph: ...
0
votes
0answers
23 views

What is the right way to set the dimension of the word representation in SkipGram word2vec?

I know that in word2vec each word has two word representations; one for the center word and one for the context word. What is the right way to set the dimension of the center word using SkipGram, ...
1
vote
0answers
30 views

Which NLP model to use to handle long context?

I'm trying to process product data for an e-commerce platform. The goal is to understand products' size. Just to show you some examples on how messy product dimension description is: ...
1
vote
0answers
35 views

Designing a chatbot personal project with zero coding experience, using an existing platform

My girlfriend has a masters degree in linguistics and would like to create an AI chatbot personal project to show potential employers her linguistics skills since she is struggling to find a job. ...
1
vote
0answers
39 views

Creating Text Features using word2vec

My task is to classify some texts. I have used word2vec to represent text words and I pass them to an LSTM as input. Taking into account that texts do not contain the same number of words, is it a ...
2
votes
0answers
63 views

Is there a good book or paper on word embeddings?

Is there a good and modern book that focuses on word embeddings and their applications? It would also be ok to provide the name of a paper that provides a good overview of word embeddings.
0
votes
0answers
22 views

How to represent labels in dialogue state tracking

In Dialogue State Tracking in most of the papers there is often a small figure that denotes how the data is structured for example Global-Locally Self-Attentive Dialogue State Tracker or Neural belief ...
3
votes
1answer
43 views

What is meant by the expected BLEU cost when training with BLEU and SIMILE?

Recently I was reading a paper based on a new evaluation metric SIMILE. In a section, validation loss comparison had been made for SIMILE and BLEU. The plot showed the expected BLEU cost when training ...
1
vote
0answers
15 views

What are the best datasets available for music information retrieval?

I am interested in doing some work in classification problems in music information retrieval. I know that there are some formats of datasets (such as MIDI, Spectrogram, Piano-roll, MusicXML, etc.) for ...
2
votes
1answer
178 views

Similarity score between 2 words using Pre-trained BERT using Pytorch

I'm trying to compare Glove, Fasttext, Bert on the basis of similarity between 2 words using Pre-trained Models. Glove and Fasttext had pre-trained models that could easily be used with gensim ...
1
vote
0answers
24 views

Building a spell check model

I have customer review texts. The data consists of the the raw and manually corrected texts of the reviews. I have aligned these pairs by using similarity algorithms and matched the words on them. ...
0
votes
0answers
24 views

dataset structure for context aware task-oriented dialogue systems

I have been reading papers on Dialog State Tracking and Intent Detection and Slot Filling for building a simple ...
1
vote
0answers
25 views

Are there any good resources (preferably books) about techniques used for entity extraction?

Given some natural language sentences like I would like to talk to Mr. Smith I would like to extract entities, like the person "Smith". I know that frameworks, which are capable of doing ...
1
vote
1answer
2k views

How to use pre-trained BERT to extract the vectors from sentences?

I'm trying to extract the vectors from the sentences. Spent soo much time searching for the pre-trained BERT models but found nothing. Is it possible to get the vectors using pre-trained BERT from ...
1
vote
0answers
516 views

GPT-2: (Hardware) requirements for fine-tuning the 774M model

I wonder if there's anyone who has actually succeeded in fine-tuning GPT-2's 774M model without using cloud TPU's. My GeForce RTX 2070 SUPER couldn't handle it in previous attempts. I'm running ...
3
votes
2answers
222 views

Levenshtein Distance between each word in a given string

From Calculate Levenshtein distance between two strings in Python it is possible to calculate distance and similarity between two given strings(sentences). And from Levenshtein Distance and Text ...
0
votes
0answers
29 views

How to Extract required fields from a text(pdf extracted)file?

I have extracted text from pdf files to each text file. Sample of a text file: ...
1
vote
0answers
26 views

How well can NLP techniques recognize connotations in natural languages?

What is the state of the art with respect to recognizing connotations in natural languages? For instance: Trump is a better president than Obama. [Praising] Trump is the worst president ...
0
votes
0answers
38 views

Best language model to do dimension description cleansing/normalization?

I'm working on a web scraper to gather product data. product dimensions are very important to me, but they come in different formats: 32w x 45h x 23d width: 32 inch height: 45 inch depth: 12 inch .....
0
votes
0answers
20 views

Keras word ordering task

I'm trying to solve the word ordering task: given a syntactically unordered sentence, recover the right order of the words. The adopted approach is to transform each sentence in a dependency tree and ...
2
votes
2answers
58 views

Could clustering be used to parse pdf documents to get headings and titles?

I'm a bit new to AI and I'd like to use some kind of clustering algorithm to solve a problem: I'm trying to parse pdf documents to get headings and titles. I can parse pdf to html and I'm then able ...
3
votes
4answers
93 views

Top Frequent occurrence word effect in Model Efficiency?

Assume that I have a Dataframe with the text column. Problem: Classification / Prediction ...
2
votes
0answers
31 views

How should I design a reward function for a NLP problem where two models interoperate?

I would like to design a reward function. I am training two models from the first model that classify set of texts (paragraphs and keywords) and I also got some hidden states. The second model is ...
1
vote
1answer
42 views

How would you build an AI to output the primary concept of a paragraph?

My thinking is you input a paragraph, or sentence, and the program can boil it down to the primary concept(s). Example: Input: Sure, it would be nice if morality was simply a navigation toward ...
7
votes
1answer
566 views

What is the intuition behind the dot product attention?

I am watching Attention all you need, In that what is the intuition behind the dot product attention? $$A(q,K, V) = \sum_i\frac{e^{q.k_i}}{\sum_j e^{q.k_j}} v_i$$ becomes: $$A(Q,K, V) = softmax(QK^...
1
vote
0answers
16 views

What challenges are there in cross-lingual entity linking?

I am reading the paper entitled Improving Candidate Generation for Low resource Cross-Lingual Entity Linking. Cross-lingual entity linking (XEL) is the task of finding referents in a target language ...
3
votes
0answers
36 views

Could zero-padding affect learning in a negative way?

I implemented an LSTM with Keras to perform word ordering task (given a syntactically unordered sentence, the goal is to label ...
4
votes
1answer
68 views

Is there any literature on the design of dialogue systems for interviews and questionnaire administration?

For my master thesis I am working on a dialogue system that should be deployed in hospitals to administer simple questionnaires to patients. I already did literature research and I'm fine with what I ...
25
votes
3answers
16k views

Why does the transformer do better than RNN and LSTM in long-range context dependencies?

I am reading the article How Transformers Work where the author writes Another problem with RNNs, and LSTMs, is that it’s hard to parallelize the work for processing sentences, since you have to ...
1
vote
1answer
37 views

Should I be balancing the data before creating the vocab-to-index dictionary?

My question is about when to balance training data for sentiment analysis. Upon evaluating my training dataset, which has 3 labels (good, bad, neutral), I noticed there were twice as many neutral ...
2
votes
1answer
87 views

What are the challenges faced by using NLP to convert mathematical texts into formal logic?

From what I've figured (a) converting mathematical theorems and proofs from English to formal logic is a straightforward job for mathematicians with sufficient background, except that it takes time. ...
2
votes
0answers
35 views

How do LSTM and GRU avoid to overcome the vanishing gradient problem?

I'm watching the video Recurrent Neural Networks (RNN) | RNN LSTM | Deep Learning Tutorial | Tensorflow Tutorial | Edureka where the author says that the LSTM and GRU architecture help to reduce the ...
2
votes
0answers
27 views

How does one detect linguistic recursion so as to know how much nesting there is, if any?

To be clear, recursion in linguistics is here better called "nesting" in this CS context to avoid confusing it with the other recursion. How does one detect nesting? I am particularly ...
1
vote
0answers
35 views

What are examples of tutorials and blogs for beginners to master the cross-lingual information retrieval?

Currently, I am following the Dan Jurofsky NLP Tutorial and CS 224 Stanford 2019. Can you list tutorials and blogs for beginners to master the cross-lingual information retrieval?
1
vote
0answers
19 views

What are the current research trends in recognizing narrative similarity?

I am currently working on a term paper on the topic of Narrative Similarity, based on Loizos Michael's work "Similarity of Narratives". I am trying to find the latest trends within this field of study ...
1
vote
0answers
75 views

Building a template based NLG system to generate a report from data

I am a newbie to NLP and NLG. I am tasked to develop a system to generate a report based on a given data table. The structure of the report and the flow is predefined. I have researched on several ...
1
vote
2answers
65 views

Is text preprocessing really all that necessary for NLP?

As a first step in many NLP courses, we learn about text preprocessing. The steps include lemmatization, removal of rare words, correcting typos etc. But I am not so sure about the actual ...

1 2 3
4
5
11