Questions tagged [natural-language-processing]

For questions related to natural language processing (NLP), which is concerned with the interactions between computers and human (or natural) languages, in particular how to create programs that process and analyze large amounts of natural language data.

Filter by
Sorted by
Tagged with
0
votes
0answers
9 views

What is the recommended way to retokenize ontonotes to wordpiece?

I am using ontonotes5 dataset which is already tokenize. But since I want to use Bert. I want to tokenize it myself, keeping all the annotations correct.
1
vote
0answers
25 views

What is the meaning of "Our current objective weights every token equally and lacks a notion of what is most important to predict" in the GPT-3 paper?

On page 34 of OpenAI's GPT-3, there is a sentence demonstrating the limitation of objective function: Our current objective weights every token equally and lacks a notion of what is most important to ...
2
votes
0answers
24 views

How to Select Model Parameters for Transformer (Heads, number of layers, etc)

Is there a general guideline on how the Transformer model parameters should be selected, or the range of these parameters that should be included in a hyperparameter sweep? Number of heads Number of ...
2
votes
0answers
62 views

Extracting "hidden" costs from financial statements using NLP

I'm designing a NLP model to extract various kinds of "hidden" expenses from 10-K and 10-Q financial statements. I've come up with about 7 different expense categories (restructuring costs, ...
0
votes
0answers
34 views

How do the trainable projection layer used in PRADO and pQRNN work?

Trainable projection layers are said to be a very powerful thing but after reading: https://www.aclweb.org/anthology/D19-1506.pdf https://arxiv.org/pdf/2101.08890.pdf I don't understand how it works....
-1
votes
1answer
28 views

Extracting values from text based on keywords

I am trying to read a PDF file and put it in Python string and trying to fetch information based on keywords. The text here is completely irregular. Example of text Blockquote Ram has taken an ...
4
votes
2answers
428 views

Why does GPT-2 Exclude the Transformer Encoder?

After looking into transformers, BERT, and GPT-2, from what I understand, GPT-2 essentially uses only the decoder part of the original transformer architecture and uses masked self-attention that can ...
0
votes
0answers
31 views

Why (not) using pre-processing before using Transformer models?

Regarding the use of pre-processing techniques before using Transformers models, I read this post that apparently says that these measures are not so necessary nor interfere so much in the final ...
0
votes
0answers
20 views

Text analysis or just statistical model?

For a college class, I have to train an ML model. I have an idea for a project but I am unsure if it is wise to attempt to solve this problem with ML as opposed to maybe more conventional methods of ...
0
votes
0answers
17 views

Extracting keywords from documents and predicting future keywords

I apologise for the newbie question! I've played with ML before, but at a very superficial level. I now have an interesting problem to solve and I wondered whether you could share some thoughts - I ...
2
votes
1answer
58 views

I have 5000 html files (structured text), how can I generate a new one that "resembles" those?

I don't know anything about ML or NLP, but I was asked by someone to create brand new statutes (written laws) that resemble the ones currently in effect in my country. I have already gathered the laws,...
0
votes
0answers
36 views

Adding BERT embeddings in BiLSTM embedding layer

I am want to use BERT embeddings in the BiLSTM embedding layer instead of Word2Vec or FastText Embeddings. There is any code to do that?
0
votes
1answer
37 views

How to train a sequence labeling model with annotations from three annotators?

I have a dataset of movie reviews annotated by 3 persons. The following example contains one sentence with corresponding annotations from 3 different persons. ...
1
vote
1answer
59 views

Can Facebook's LASER be used like BERT?

Can Facebook's LASER be fine-tuned like BERT for Question Answering tasks or Sentiment Analysis? From my understanding, they created an embedding that allows for similar words in different languages ...
0
votes
2answers
94 views

Is my approach to building an RNN to predict the probability that the word is in English appropriate?

Goal To build an RNN which would receive a word as an input, and output the probability that the word is in English (or at least would be English sounding). Example ...
2
votes
3answers
156 views

Is there a relationship between Computer Algebra and NLP?

My intuition is that there is some overlap between understanding language and symbolic mathematics (e.g. algebra). The rules of algebra are somewhat like grammar, and the step-by-step arguments get ...
0
votes
2answers
169 views

Should I need to use BERT embeddings while tokenizing using BERT tokenizer?

I am new to BERT and NLP and I am a little confused with tokenization and word embedding. My doubt is if I use the BertTokenizer for tokenizing a sentence then do I have to compulsorily use ...
0
votes
0answers
9 views

Is Universal Sentence Encoder helping producing supervised or not summaries?

I am currently working on generating automatic summaries of scientific texts and am wondering whether using Google's Universal Sentence Encoder makes my approach data-driven or supervised. I am doing ...
2
votes
1answer
33 views

Are training sequences for LMs sampled in an IID fashion?

If I understand correctly, when training language models, we take a document and then chunk the document into a sequences of k tokens. So if the document is of length 30 and k=10, then we'll have 20 ...
0
votes
0answers
22 views

How does BERT answer questions?

I have been trying to understand how the BERT model works. Specifically, I am trying to understand how it picks up answers to questions on a given passage. I have tried following this blog post and, ...
0
votes
0answers
29 views

How does Byte Pair Embedding affect the effectiveness of embedding techniques like Word2Vec?

I see Byte Pair Embedding (BPE) is frequently adopted for NLP to reduce the vocabulary size. How does it affect the effectiveness of embedding techniques like Word2Vec?
3
votes
2answers
1k views

What is the difference between a language model and a word embedding?

I am self-studying applications of deep learning on the NLP and machine translation. I am confused about the concepts of "Language Model", "Word Embedding", "BLEU Score". ...
0
votes
0answers
27 views

Is there an effective way of obtaining the topic distribution for a given document from a VAE-LDA?

Is there an effective way of obtaining the topic distribution for a given document from a Variational AutoEncoder Latent Dirichlet Allocation (VAE-LDA)? Most existing public VAE-LDA codebases seem to ...
1
vote
0answers
24 views

Given the word embeddings, how do I create the sentence composed of the corresponding words?

I have done some reading. I want to implement an LSTM with pre-trained word embeddings (I also have plans to create my word embeddings, but let's cross that bridge when we come to it). In any given ...
1
vote
1answer
52 views

CAPTCHA based on text comprehension and random tokens

I developed a novel type of CAPTCHA based on text comprehension and random tokens. Given a task Pick the first pair of adjacent letters and a random token ...
0
votes
0answers
68 views

Understanding Huggingface QA Transformers Confidence Scores

I seem to be unable to find information on whether the confidence scores for the huggingface transformers used for QA are in the form of 10^-2 or 10^-5. What are your thoughts on the matter? Example: <...
3
votes
0answers
51 views

What is the difference between zero-padding and character-padding in Recurrent Neural Networks?

For RNN's to work efficiently, we vectorize the operations, which results in an input matrix of shape (m, max_seq_len) where m ...
2
votes
1answer
140 views

How are certain machine learning models able to produce variable-length outputs given variable-length inputs?

Most machine learning models, such as multilayer perceptrons, require a fixed-length input and output, but generative (pre-trained) transformers can produce sentences or full articles of variable ...
2
votes
0answers
24 views

Adding corpus to BERT for QA

I was wondering about SciBERT's QA abilities using SQuAD. I have a scarce textual dataset consisting of less than 100 files where doctors are discussing cancer in dialogues. I want to add it to ...
0
votes
1answer
34 views

Are the Word2Vec encoded embeddings available online? [closed]

I am trying to do an NLP project and was wondering if there is anywhere online where the Word2Vec embeddings are stored (the actual n-dimmensional vectors). I want to search up a word and see what its ...
0
votes
1answer
36 views

NLP: Are hashtags tokenised?

I am exploring a potential NLP project. I was wondering what generally is done with the hashtags words (e.g. #hello). Are those words ignored? is the ...
1
vote
1answer
40 views

How to keep track of the subject/entity in a sentence?

I'm working on Sentiment Analysis, using HuggingFace to perform sentiment analysis on articles ...
3
votes
1answer
76 views

How would one disambiguate between two meanings of the same word in a sentence?

The boy lifted the bat and hit the ball. In the above sentence, the noun "bat" means the wooden stick. It does not mean bat, the flying mammal, which is also a noun. Using NLP libraries to ...
1
vote
0answers
35 views

What does the outputlayer of BERT for masked language modelling look like?

In the tutorial BERT – State of the Art Language Model for NLP the masked language modeling pre-training steps are described as follows: In technical terms, the prediction of the output words ...
5
votes
3answers
965 views

What kind of word embedding is used in the original transformer?

I am currently trying to understand transformers. To start, I read Attention Is All You Need and also this tutorial. What makes me wonder is the word embedding used in the model. Is word2vec or GloVe ...
0
votes
0answers
35 views

Is there a reason why no one combines word embeddings with the median?

Could you combine word embeddings with the median per dimension to get a document embedding? In my case I have a huge amount of words to build one document, which in turn should describe a topic. I ...
1
vote
0answers
27 views

How to scrape product data on supplier websites?

I'm currently trying to build a semantic scraper that can extract product information from different company websites of suppliers in the packaging industry (with as little manual customization per ...
0
votes
0answers
123 views

Is using a LSTM, CNN or any other neural network model on top of a Transformer(using hidden states) overkill?

I have recently come across transformers, I am new to Deep Learning. I have seen a paper using CNN and BiLSTM on top of a transformer, the paper uses a transformer(XLM-R) for sentiment analysis in ...
0
votes
1answer
169 views

Is the working of RNNs, LSTM and GRU sequential or parallel?

You take any blog or any example and all they tell you about is the given picture below. It has 4 different matrices and 3 of whose weights are shared. So, I'm wondering how is this achieved in ...
0
votes
0answers
16 views

Is there any neural network model that can perform multiple NLP steps at once?

I realize most NLP algorithms have multiple steps. (e.g. OCR/speech rec > syntax > semantics > response logic > semantic output > natural language output) Is there any NN model that can ...
1
vote
0answers
24 views

What is the best approach for sentiment analysis when the text is very brief?

I'm working on a project to do sentiment analysis but my data is not long and properly formatted text. It's more likely to be very short sentences, e.g. tweets (in full tweet lingo), quick reviews of ...
1
vote
1answer
117 views

What dataset might Elon Musk's Dall-E have used?

Dall-E, it can generate many imaginative images from the description, even some peculiar images, how did they actually create this kind of dataset to train this AI , because there is not much of that ...
1
vote
2answers
131 views

How to extract parameters from a text using AI/NLP

lets say I have three texts: "make a heading that says hello word" "make a heading of hello world" "create heading consist of hello world" How can I fetch those groups ...
2
votes
1answer
55 views

How can I find words in a string that are related to a given word, then associate a sentiment to that found word?

I came up with an NLP-related problem where I have a list of words and a string. My goal is to find any word in the list of words that is related to the given string. Here is an example. Suppose a ...
0
votes
0answers
24 views

Is there a reference that describes Recurrent Neural Networks for NLP tasks?

I would like some references of works that try to understand the functioning of any kind of RNN in natural language processing tasks. They can be any work that tries to explain the functioning of the ...
3
votes
1answer
2k views

How to understand 'losses' in Spacy's custom NER training engine?

From the tid-bits, I understand of neural networks (NN), the Loss function is the difference between predicted output and expected output of the NN. I am following this tutorial, the losses are ...
2
votes
0answers
132 views

T5 or BERT for sentence correction/generation task?

I have sentences with some grammatical errors , with no punctuations and digits written in words... something like below: As you can observe, a proper noun , winston isnt highlighted with capital in ...
0
votes
1answer
86 views

Should we use a pre-trained model or a blank model for custom entity training of NER in spacy?

Further to my last question, I am training a custom entity of FOODITEM to be recognized by Spacy's Name Entity Recognition engine. I am following tutorials online, following is the advise given in ...
1
vote
0answers
23 views

Is there any research work that shows that we should explicitly mark the word boundaries for 1D CNNs?

I'm doing character embedding for NLP tasks using one-dimensional convolutional neural networks (see Chiu and Nichols (2016) for the motivation). I haven't found any empirical evidence of whether or ...
0
votes
0answers
39 views

Time series analysis using computer vision principles

I'm just starting to explore topics within computer vision and curious if there are any concepts in that area that could be applied to segmenting multivariate time series with the goal of grouping ...

1 2
3
4 5
12