Questions tagged [natural-language-processing]

For questions related to natural language processing (NLP), which is concerned with the interactions between computers and human (or natural) languages, in particular how to create programs that process and analyze large amounts of natural language data.

Filter by
Sorted by
Tagged with
99 votes
5 answers
84k views

How can neural networks deal with varying input sizes?

As far as I can tell, neural networks have a fixed number of neurons in the input layer. If neural networks are used in a context like NLP, sentences or blocks of text of varying sizes are fed to a ...
Asciiom's user avatar
  • 1,141
71 votes
4 answers
96k views

Why does the transformer do better than RNN and LSTM in long-range context dependencies?

I am reading the article How Transformers Work where the author writes Another problem with RNNs, and LSTMs, is that it’s hard to parallelize the work for processing sentences, since you have to ...
DRV's user avatar
  • 1,673
53 votes
2 answers
45k views

How does ChatGPT retain the context of previous questions?

One of the innovations with OpenAI's ChatGPT is how natural it is for users to interact with it. What is the technical enabler for ChatGPT to maintain the context of previous questions in its answers? ...
milez's user avatar
  • 971
35 votes
2 answers
19k views

How can Transformers handle arbitrary length input?

The transformer, introduced in the paper Attention Is All You Need, is a popular new neural network architecture that is commonly viewed as an alternative to recurrent neural networks, like LSTMs and ...
chessprogrammer's user avatar
34 votes
6 answers
11k views

How does an AI like ChatGPT answer a question in a subject which it may not know?

After seeing StackOverflow's banning of ChatGPT, I explored it out of curiosity. It's marvellous as it can write code by itself! Later to check if it knows chess as well like Google-Deepmind's ...
iammilind's user avatar
  • 443
31 votes
3 answers
41k views

Can BERT be used for sentence generating tasks?

I am a new learner in NLP. I am interested in the sentence generating task. As far as I am concerned, one state-of-the-art method is the CharRNN, which uses RNN to generate a sequence of words. ...
ch271828n's user avatar
  • 413
30 votes
9 answers
7k views

What is the actual quality of machine translations?

As an AI layman, till today I am confused by the promised and achieved improvements of automated translation. My impression is: there is still a very, very far way to go. Or are there other ...
Hans-Peter Stricker's user avatar
28 votes
4 answers
35k views

Why is ChatGPT bad at math?

As opposed to How does ChatGPT know math?, I've been seeing some things floating around the Twitterverse about how ChatGPT can actually be very bad at math. For instance, I asked it "If it takes ...
Mithical's user avatar
  • 2,895
26 votes
1 answer
15k views

How is BERT different from the original transformer architecture?

As far as I can tell, BERT is a type of Transformer architecture. What I do not understand is: How is Bert different from the original transformer architecture? What tasks are better suited for BERT,...
chessprogrammer's user avatar
24 votes
5 answers
11k views

Why does ChatGPT fail in playing "20 questions"?

IBM Watson's success in playing "Jeopardy!" was a landmark in the history of artificial intelligence. In the seemingly simpler game of "Twenty questions" where player B has to ...
Hans-Peter Stricker's user avatar
20 votes
1 answer
10k views

What is the intuition behind the dot product attention?

I am watching the video Attention Is All You Need by Yannic Kilcher. My question is: what is the intuition behind the dot product attention? $$A(q,K, V) = \sum_i\frac{e^{q.k_i}}{\sum_j e^{q.k_j}} v_i$$...
DRV's user avatar
  • 1,673
20 votes
2 answers
19k views

What are the main differences between skip-gram and continuous bag of words?

The skip-gram and continuous bag of words (CBOW) are two different types of word2vec models. What are the main differences between them? What are the pros and cons of both methods?
DRV's user avatar
  • 1,673
20 votes
2 answers
12k views

Why does GPT-2 Exclude the Transformer Encoder?

After looking into transformers, BERT, and GPT-2, from what I understand, GPT-2 essentially uses only the decoder part of the original transformer architecture and uses masked self-attention that can ...
Athena Wisdom's user avatar
18 votes
3 answers
11k views

What kind of word embedding is used in the original transformer?

I am currently trying to understand transformers. To start, I read Attention Is All You Need and also this tutorial. What makes me wonder is the word embedding used in the model. Is word2vec or GloVe ...
Bert Gayus's user avatar
18 votes
2 answers
364 views

What research has been done in the domain of "identifying sarcasm in text"?

Identifying sarcasm is considered one of the most difficult open-ended problems in the domain of ML and NLP/NLU. So, was there any considerable research done on that front? If yes, then what is the ...
Dawny33's user avatar
  • 1,371
17 votes
3 answers
1k views

How would an AI learn language?

I was think about AIs and how they would work, when I realised that I couldn't think of a way that an AI could be taught language. A child tends to learn language through associations of language and ...
AvahW's user avatar
  • 275
16 votes
1 answer
23k views

What language is the GPT-3 engine written in?

I know that the API is python based, but what's the gpt-3 engine written in mostly? C? C++? I'm having some trouble finding this info.
Otherness's user avatar
  • 285
16 votes
3 answers
2k views

What roles knowledge bases play now and will play in the future?

Nowadays, artificial intelligence seems almost equal to machine learning, especially deep learning. Some have said that deep learning will replace human experts, traditionally very important for ...
Lerner Zhang's user avatar
15 votes
2 answers
7k views

Why does ChatGPT not give the answer text all at once?

When ChatGPT is generating an answer to my question, it generates it word by word. So I actually have to wait until I get the final answer. Is this just for show? Or is it really real-time generating ...
Sander van den Oord's user avatar
14 votes
1 answer
10k views

Are the dialogs at Sophia's (the robot) appearings scripted?

I talk about the robot from: Hanson Robotics, which was granted the right to citizenship from Saudi Arabia. I have found the following articles: Your new friend is a humanoid robot Source: ...
tgogos's user avatar
  • 251
14 votes
3 answers
2k views

What are the specific requirements of the Turing test?

What are the specific requirements of the Turing test? What requirements if any must the evaluator fulfill in order to be qualified to give the test? Must there always be two participants in the ...
Luke's user avatar
  • 253
14 votes
3 answers
18k views

How do I compute the structural similarity between sentences?

I am working on a problem where I need to determine whether two sentences are similar or not. I implemented a solution using BM25 algorithm and wordnet synsets for determining syntactic & ...
Shubham Tiwari's user avatar
13 votes
6 answers
520 views

Have any AI systems yet been developed that can knowingly lie to / deceive a human?

AI systems today are very capable machines, and recently the area of Natural Language Processing and Response has been exploding with innovation, as well as the fundamental algorithmic structure of AI ...
Avik Mohan's user avatar
13 votes
1 answer
2k views

Can an AI make a constructed (natural) language?

According to Wikipedia An artificial or constructed language (sometimes called a conlang) is a language that has been created by a person or small group, instead of being formed naturally as part of ...
Kevin's user avatar
  • 233
12 votes
2 answers
691 views

Is anybody still using Conceptual Dependency Theory?

Roger Schank did some interesting work on language processing with Conceptual Dependency (CD) in the 1970s. He then moved somewhat out of the field, being in Education these days. There were some ...
Oliver Mason's user avatar
  • 5,397
12 votes
1 answer
483 views

How can Viv generate new code based on some user's query?

I have been looking into Viv, an artificial intelligent agent in development. Here is a demonstration of Viv (by Dag Kittlaus). Based on what I understand, this AI can generate new code and execute it ...
N. Chalifour's user avatar
11 votes
3 answers
8k views

What is the purpose of Decoder mask (triangular mask) in Transformer?

I'm trying to implement transformer model using this tutorial. In the decoder block of the Transformer model, a mask is passed to "pad and mask future tokens in the input received by the decoder&...
Uchiha Madara's user avatar
11 votes
4 answers
5k views

Is there any open source counterpart to the IBM Watson? [closed]

I am looking for something similar to IBM Watson but open source.
arash moradi's user avatar
11 votes
2 answers
12k views

What are the segment embeddings and position embeddings in BERT?

They only reference in the paper that the position embeddings are learned, which is different from what was done in ELMo. ELMo paper - https://arxiv.org/pdf/1802.05365.pdf BERT paper - https://arxiv....
Skinish's user avatar
  • 163
11 votes
1 answer
5k views

What is the intuition behind the attention mechanism?

Attention idea is one of the most influential ideas in deep learning. The main idea behind attention technique is that it allows the decoder to "look back” at the complete input and extracts ...
Pluviophile's user avatar
  • 1,263
10 votes
2 answers
7k views

What is different in each head of a multi-head attention mechanism?

I have a difficult time understanding the "multi-head" notion in the original transformer paper. What makes the learning in each head unique? Why doesn't the neural network learn the same ...
mhsnk's user avatar
  • 123
10 votes
3 answers
889 views

Is there an AI model for generating sarcastic sentences?

I am currently working on an Android AI app. I am aware of AI models to generate random sentences. However, is there an AI model for generating sarcastic sentences?
Praharsh Bhatt's user avatar
10 votes
1 answer
310 views

Has anyone attempted to train an AI to learn all languages?

It seems that most projects attempt to teach the AI to learn individual, specific languages. It occurs to me that there are relations in written and spoken words and phrases across languages - most ...
mindplay.dk's user avatar
9 votes
4 answers
1k views

Can the English language ever be generalized using a set of grammar rules?

In programming languages, there is a set of grammar rules which govern the construction of valid statements and expressions. These rules help in parsing the programs written by the user. Can there ...
skrtbhtngr's user avatar
9 votes
3 answers
5k views

Can AI write good jokes yet?

Just watched a recent WIRED video on virtual assistants' performance on telling jokes. They're composed by humans, but I'd like to know if AI has gotten good enough to write some.
user avatar
8 votes
4 answers
5k views

Why does this multiplication of $Q$ and $K$ have a variance of $d_k$, in scaled dot product attention?

In scaled dot product attention, we scale our outputs by dividing the dot product by the square root of the dimensionality of the matrix: The reason why is stated that this constrains the ...
Jacob B's user avatar
  • 247
8 votes
4 answers
415 views

Is language understanding possible without qualia?

I feel that many words if not all of them have a direct mapping to some kind of inner subjective experience, to a physical object, mental feeling, process or some other kind of abstract thing. Given ...
Vivek Krishna's user avatar
8 votes
1 answer
188 views

What artificial intelligence strategies are useful for summarization?

If I have a paragraph I want to summarize, for example: Ponzo and Fila went to the mall during the day. They walked for a long while, stopping at shops. They went to many shops. At first, they didn'...
dynrepsys's user avatar
  • 1,363
8 votes
3 answers
3k views

Have capsule neural networks been used to NLP problems?

The capsule neural network seems to be a good solution for problems that involve hierarchies. For example, a face is composed of eyes, a nose and ears; a hand is made of fingers, nails, and a palm; ...
Ahti Ahde's user avatar
  • 278
8 votes
2 answers
6k views

How is the F1 score calculated in a question-answering system?

I have an NLP model for answer-extraction. So, basically, I have a paragraph and a question as input, and my model extracts the span of the paragraph that corresponds to the answer to the question. I ...
HLeb's user avatar
  • 579
8 votes
2 answers
208 views

Are there any algorithms that can measure the concept similarity between texts?

Are there any algorithms (or software libraries) that can be used to detect the similarity of concepts in text, regardless of articulation, grammar, synonyms, etc.? For example, these phrases: ...
omega's user avatar
  • 189
8 votes
2 answers
550 views

How can I translate a natural language question to an MDX query?

I am researching Natural Language Processing (NLP) to develop an NL Question Answering system. The answering part is already done. So processing the question remains, along with the questions ...
lilienfa's user avatar
  • 319
8 votes
2 answers
1k views

What are the current big challenges in natural language processing and understanding?

I'm doing a paper for a class on the topic of big problems that are still prevalent in AI, specifically in the area of natural language processing and understanding. From what I understand, the areas: ...
Landon G's user avatar
  • 500
8 votes
1 answer
191 views

Why are documents kept separated when training a text classifier?

Most of the literature considers text classification as the classification of documents. When using the bag-of-words and Bayesian classification, they usually use the statistic TF-IDF, where TF ...
freesoul's user avatar
  • 246
8 votes
2 answers
2k views

Why is the perplexity a good evaluation metric for chatbots?

A few papers I have come across say that BLEU is not an appropriate evaluation metric for chatbots, so they use the perplexity. First of all, what is perplexity? How to calculate it? And why is ...
RuiZhang1993's user avatar
8 votes
1 answer
813 views

Are there transformer-based architectures that can produce fixed-length vector encodings given arbitrary-length text documents?

BERT encodes a piece of text such that each token (usually words) in the input text map to a vector in the encoding of the text. However, this makes the length of the encoding vary as a function of ...
HelloGoodbye's user avatar
7 votes
2 answers
1k views

How would an AI learn idiomatic phrases in a natural language?

After an AI goes through the process described in How would an AI learn language?, an AI knows the grammar of a language through the process of grammar induction. They can speak the language, but they ...
Marvin's user avatar
  • 83
7 votes
2 answers
208 views

What was the first machine that was able to carry on a conversation?

What was the first AI that was able to carry on a conversation, with real responses, such as in the famous 'I am not a robot. I am a unicorn' case? A 'real response' constitutes a sort-of ...
Mithical's user avatar
  • 2,895
7 votes
2 answers
620 views

How to determine if an Amazon review is likely to be fake using text classification

I'm currently in the research stage of building a web app in ASP.NET where the user can input a URL to an Amazon product, then the app would determine how likely its reviews are to be genuine. I need ...
user8709398's user avatar
7 votes
1 answer
2k views

Did Turing foresee the required capabilities to pass the Turing test?

In Section 1.1 of Artificial Intelligence: A Modern Approach, it is stated that a computer which passes the Turing Test would need 4 capabilities, and that these 4 capabilities comprise most of the ...
alwaysLearningABC's user avatar

1
2 3 4 5
16