Questions tagged [natural-language-processing]

For questions related to natural language processing (NLP), which is concerned with the interactions between computers and human (or natural) languages, in particular how to create programs that process and analyze large amounts of natural language data.

Filter by
Sorted by
Tagged with
28 votes
4 answers
35k views

Why is ChatGPT bad at math?

As opposed to How does ChatGPT know math?, I've been seeing some things floating around the Twitterverse about how ChatGPT can actually be very bad at math. For instance, I asked it "If it takes ...
Mithical's user avatar
  • 2,895
99 votes
5 answers
84k views

How can neural networks deal with varying input sizes?

As far as I can tell, neural networks have a fixed number of neurons in the input layer. If neural networks are used in a context like NLP, sentences or blocks of text of varying sizes are fed to a ...
Asciiom's user avatar
  • 1,141
15 votes
2 answers
7k views

Why does ChatGPT not give the answer text all at once?

When ChatGPT is generating an answer to my question, it generates it word by word. So I actually have to wait until I get the final answer. Is this just for show? Or is it really real-time generating ...
Sander van den Oord's user avatar
35 votes
2 answers
19k views

How can Transformers handle arbitrary length input?

The transformer, introduced in the paper Attention Is All You Need, is a popular new neural network architecture that is commonly viewed as an alternative to recurrent neural networks, like LSTMs and ...
chessprogrammer's user avatar
6 votes
1 answer
2k views

How was ChatGPT trained?

I know that large language models like GPT-3 are trained simply to continue pieces of text that have been scraped from the web. But how was ChatGPT trained, which, while also having a good ...
HelloGoodbye's user avatar
71 votes
4 answers
96k views

Why does the transformer do better than RNN and LSTM in long-range context dependencies?

I am reading the article How Transformers Work where the author writes Another problem with RNNs, and LSTMs, is that it’s hard to parallelize the work for processing sentences, since you have to ...
DRV's user avatar
  • 1,673
53 votes
2 answers
45k views

How does ChatGPT retain the context of previous questions?

One of the innovations with OpenAI's ChatGPT is how natural it is for users to interact with it. What is the technical enabler for ChatGPT to maintain the context of previous questions in its answers? ...
milez's user avatar
  • 971
24 votes
5 answers
11k views

Why does ChatGPT fail in playing "20 questions"?

IBM Watson's success in playing "Jeopardy!" was a landmark in the history of artificial intelligence. In the seemingly simpler game of "Twenty questions" where player B has to ...
Hans-Peter Stricker's user avatar
17 votes
3 answers
1k views

How would an AI learn language?

I was think about AIs and how they would work, when I realised that I couldn't think of a way that an AI could be taught language. A child tends to learn language through associations of language and ...
AvahW's user avatar
  • 275
16 votes
3 answers
2k views

What roles knowledge bases play now and will play in the future?

Nowadays, artificial intelligence seems almost equal to machine learning, especially deep learning. Some have said that deep learning will replace human experts, traditionally very important for ...
Lerner Zhang's user avatar
14 votes
3 answers
18k views

How do I compute the structural similarity between sentences?

I am working on a problem where I need to determine whether two sentences are similar or not. I implemented a solution using BM25 algorithm and wordnet synsets for determining syntactic & ...
Shubham Tiwari's user avatar
5 votes
2 answers
2k views

Is the Mask Needed for Masked Self-Attention During Inference with GPT-2

My understanding is that masked self-attention is necessary during training of GPT-2, as otherwise it would be able to directly see the correct next output at each iteration. My question is whether ...
D_s's user avatar
  • 51
2 votes
2 answers
64 views

Are any AI systems available, or in development, for finding and analysing fallacious inference in natural language text?

Poor reasoning, and ignorance in general, is the source of a lot of suffering and evil. Covertly erroneous logic is often used in manipulation. And much of this broken thought is being used directly ...
Michael's user avatar
  • 146
30 votes
9 answers
7k views

What is the actual quality of machine translations?

As an AI layman, till today I am confused by the promised and achieved improvements of automated translation. My impression is: there is still a very, very far way to go. Or are there other ...
Hans-Peter Stricker's user avatar
20 votes
2 answers
12k views

Why does GPT-2 Exclude the Transformer Encoder?

After looking into transformers, BERT, and GPT-2, from what I understand, GPT-2 essentially uses only the decoder part of the original transformer architecture and uses masked self-attention that can ...
Athena Wisdom's user avatar
18 votes
3 answers
11k views

What kind of word embedding is used in the original transformer?

I am currently trying to understand transformers. To start, I read Attention Is All You Need and also this tutorial. What makes me wonder is the word embedding used in the model. Is word2vec or GloVe ...
Bert Gayus's user avatar
14 votes
3 answers
2k views

What are the specific requirements of the Turing test?

What are the specific requirements of the Turing test? What requirements if any must the evaluator fulfill in order to be qualified to give the test? Must there always be two participants in the ...
Luke's user avatar
  • 253
14 votes
1 answer
10k views

Are the dialogs at Sophia's (the robot) appearings scripted?

I talk about the robot from: Hanson Robotics, which was granted the right to citizenship from Saudi Arabia. I have found the following articles: Your new friend is a humanoid robot Source: ...
tgogos's user avatar
  • 251
11 votes
3 answers
8k views

What is the purpose of Decoder mask (triangular mask) in Transformer?

I'm trying to implement transformer model using this tutorial. In the decoder block of the Transformer model, a mask is passed to "pad and mask future tokens in the input received by the decoder&...
Uchiha Madara's user avatar
8 votes
4 answers
5k views

Why does this multiplication of $Q$ and $K$ have a variance of $d_k$, in scaled dot product attention?

In scaled dot product attention, we scale our outputs by dividing the dot product by the square root of the dimensionality of the matrix: The reason why is stated that this constrains the ...
Jacob B's user avatar
  • 247
7 votes
1 answer
2k views

Did Turing foresee the required capabilities to pass the Turing test?

In Section 1.1 of Artificial Intelligence: A Modern Approach, it is stated that a computer which passes the Turing Test would need 4 capabilities, and that these 4 capabilities comprise most of the ...
alwaysLearningABC's user avatar
5 votes
2 answers
224 views

What is the intuition behind how word embeddings bring information to a neural network?

How is it that a word embedding layer (say word2vec) brings more insights to the neural network compared to a simple one-hot encoded layer? I understand how the word embedding carries some semantic ...
silkAdmin's user avatar
  • 209
4 votes
1 answer
1k views

How do Transformer decoders handle arbitrary length input?

I am working through a Tensorflow Neural Machine Translation tutorial (https://www.tensorflow.org/text/tutorials/transformer) and am confused about how the decoder handles inputs when making ...
Dylan Larrabee's user avatar
4 votes
1 answer
502 views

How could you generate sentences from lists of facts

Let's pretend we had a list of facts (similar to prolog tuples) that define some knowledge about some entities. e.g. ...
Jasper Lyons's user avatar
3 votes
0 answers
97 views

What is the difference between zero-padding and character-padding in Recurrent Neural Networks?

For RNN's to work efficiently, we vectorize the operations, which results in an input matrix of shape (m, max_seq_len) where m ...
PhysicsMan's user avatar
3 votes
1 answer
55 views

How much the dialects recognition and speech recognition are relevant?

In this tutorial, they build a speech recognition model to classify a one-second audio clip as one of ten predefined words. Suppose that we modified this problem as the following: Given an Arabic ...
Abdulkader's user avatar
3 votes
0 answers
113 views

Is there a good book or paper on word embeddings?

Is there a good and modern book that focuses on word embeddings and their applications? It would also be ok to provide the name of a paper that provides a good overview of word embeddings.
ddaedalus's user avatar
  • 919
2 votes
0 answers
262 views

How do the sine and cosine functions encode position in the transformer?

After going through both the "Illustrated Transformer" and "Annotated Transformer" blog posts, I still don't understand how the sinusoidal encodings are representing the position of elements in the ...
shoshi's user avatar
  • 121
1 vote
1 answer
719 views

How is ChatGPT trained?

According to OpenAI, ChatGPT is trained in a 3-step process. Are the steps where human AI trainers are involved, i.e. training the initial policy and providing the A>B>C>D grading as ...
Meatball Princess's user avatar
1 vote
1 answer
459 views

How does chatGPT know it's an AI?

I've tried several prompts to understand how does it "know" that it's an AI, but it's answers are inconclusive for me. It says that it was hard-coded to recognize keywords in this regard, ...
Jose Miguel Cruz y Celis's user avatar
1 vote
0 answers
91 views

Why is the sample size of stochastic gradient descent a power of 2?

I watched the video lecture of cs224: Stanford CS224N: NLP with Deep Learning | Winter 2019 | Lecture 2 – Word Vectors and Word Senses. They take the sample size of the window to be $2^5 = 32$ or $2^6 ...
DRV's user avatar
  • 1,673
1 vote
1 answer
786 views

How to design a NLP algorithm to find a food item in menu card list?

I am new to NLP and AI in general. I am just expecting springboard information so that I can skip all the introduction to NLP websites. I have just started studying NLP and want to know how to go ...
The White Cloud's user avatar
1 vote
0 answers
30 views

Is there any dataset to convert text to sign language? [closed]

I'm going to start working on one university project and I would like to ask a question regarding it. My project is about "Sign language synthesis from NLP" and I need to develop an ...
Nijat Mursali's user avatar
0 votes
1 answer
377 views

In the attention mechanism, why don't we normalize after multiplying values?

As this question says: In scaled dot product attention, we scale our outputs by dividing the dot product by the square root of the dimensionality of the matrix: The reason why is stated that this ...
Peyman's user avatar
  • 564
0 votes
0 answers
32 views

Understanding how continuous bag of words method learns embedded representations

I'm reading notes on word vectors here. Specifically, I'm referring to section 4.2 on page 7. First, regarding points 1 to 6 - here's my understanding: If we have a vocabulary $V$, the naive way to ...
Shirish Kulhari's user avatar
0 votes
1 answer
42 views

If the unigram precision is (N-1)/N, then the bigram precision is :

Consider the following machine translation scenario. The reference translation has N words (do not consider sentence beginner ‘hat’ and sentence finisher ‘dot’). The machine output also has N words. ...
Geeklovenerds's user avatar