Questions tagged [natural-language-processing]
For questions related to natural language processing (NLP), which is concerned with the interactions between computers and human (or natural) languages, in particular how to create programs that process and analyze large amounts of natural language data.
730
questions
0
votes
0
answers
17
views
ChatGPT and the P-NP Problem
According to ZDNet, it is an open question whether a transformer LLM like ChatGPT can facilitate the determination of a solution to the P-NP Problem. (See Can generative AI solve computer science's ...
- 101
0
votes
1
answer
17
views
How to train a seq2seq model to rephrase input text following given rules
I want to train (fine-tune) a seq2seq model to perform the task of rephrasing input following these rules :
1- always follow the pattern "Entity Verb Entity"
2- only use simple sentences : ...
0
votes
1
answer
26
views
How to get Llama-2 Rotary Embeddings?
I want to get the Llama-2 rotary embeddings. I do print(model) and get the following output:
In the picture I highlight the rotary embeddings.
How can get the ...
- 101
1
vote
1
answer
24
views
Data preparation for NLP model
I have data from our ticketing system. Currently using OpenNLP to create different models.
For simplicity I have a 10k ticket's text as category final queue of the ticket.
My questions:
Is it ...
- 111
0
votes
0
answers
6
views
Best prompt and model for fact-checking a text (disinformation/fake-news detection) [migrated]
Given a short text <text_to_check>, I want the LLM to check whether there are some facts stated in the text which are NOT true. So I want to detect 'disinformation' / 'fake news'. And the LLM ...
0
votes
1
answer
33
views
Why do current language models no longer generate to long or short texts?
One of the biggest strengths of ChatGPT is that it generates fitting text with respect to the input query. It usually stays on topic, anwers the question completely and especially does not start ...
- 1
0
votes
0
answers
19
views
NLP for classifying a YES or NO response to a question
I'm currently working on a project that requires some feature extraction. The data I have is text and comes from an interview. The interviewer asks a question, the client responds, and the interviewer ...
- 101
1
vote
1
answer
40
views
AI: Inverse questions answering - quiz style: Verify descriptive answers to a static question
I'm exploring LLMs for educative purposes an came across the topic question answering, e. g. building a system that ingests documents like PDFs and is able to answer questions about its content.
My ...
0
votes
1
answer
35
views
Keywords extractions from short names (table and column names) [closed]
I extract keywords that are table and column names (around 100,000 in the test).
I process them in Python, and as a result, I get a CSV file with sample keywords:
db_id
type
object_id
keyword
1
a
...
- 3
2
votes
0
answers
50
views
Why is an encoder + decoder model with L by L layers the same speed as as decoder only model with 2 L layers?
I was watching this lecture: https://youtu.be/27rNqGrTdSI?t=2295
In it the presenter stated that:
"An encoder + decoder model with L by L layers is actually the same speed as as decoder only ...
- 21
0
votes
1
answer
65
views
Fine Tuning a Bert Transformer. How to label for emotions and train large scripts?
From what I have seen you can fine tune a Bert model to detect emotions by labelling single sentences.
But if the text you want to evaluate is a large script with many sentences, do I need to split ...
- 53
0
votes
1
answer
21
views
How to label missing/default values for a named entity recognition dataset
I am building the training dataset for a named entity recognition model, with 2 tags: Name and Category and I am using a pre-...
- 3
0
votes
1
answer
32
views
How can BERT/Transformer models accept input batches of different sizes?
I understand that all inputs in a batch need to be of the same size. However, it seems BERT/Transformers models can accept batches with different sizes as input.
How is that possible? I thought we ...
- 1
0
votes
1
answer
33
views
How word2vec de-embeds the special names in language models which output text
I am new to nlp field. I have some questions about word2vec embeddings. as I know they have a fixed size dictionary of vocabs. so definitely there some words which is not in that predefined dictionary ...
- 101
0
votes
1
answer
31
views
How does GPT like Decoder only conversational models distunguish the source of text?
In a conversational setting where two sources of text (user and the model) follow each other like below
User: some text bla bla
Model: another text bah bah
User: bla bla bla
Model: bah bah
and so on, ...
- 296
0
votes
0
answers
28
views
Concatenation of Feature vectors in transformers before passing to fcnn
** As I am new to the field , the question might feel little abstract and naïve considering my experience.
I am studying the Transformer architecture and trying to understand the various components ...
0
votes
0
answers
19
views
Create samples out of documents for Causal Language Modelling
I want to create an input source for Causal Language model using Llama 2 model in hugging face. I have a set of documents which are scraped from a specific website and want to fine-tune on them. Each ...
- 1
0
votes
0
answers
14
views
Has anyone tried to derive linguistic information from GPT internals?
In Stephen Wolfram's write up on the workings of GPT, he suggests that chat GPT may have identified invariant rules of human language that haven't been formalized yet, ie new linguistic findings, and ...
- 195
0
votes
0
answers
20
views
Any research in "probe-tuning" of LLMs?
Is there any research in "probe-tuning" of LLMs, i.e., tuning LLM's parameter weights such that a specific probe (classifier) is more reliably detecting certain markers throughout the ...
- 101
0
votes
0
answers
14
views
What if in DPR (dense passage retrieval), the answer belongs to more than one passage?
In the DPR paper
the dataset is expected to be in this format D = {<qi, pi+, pi,1-, ... >}
With only one positive passage, but it is possible that the question requires an answer that spans ...
- 1
0
votes
0
answers
170
views
Optimal Quantity of Training Data for Fine-Tuning an LLM: Is Bigger Always Better?
I am currently working on fine-tuning an LLM for a specific task, and I am trying to determine the optimal size for my training dataset. Intuitively, one might think that the more data, the better. ...
- 544
0
votes
1
answer
13
views
Seq2Seq model- Confusing about the dimension of Seq2Seq model [closed]
I am new to Seq2Seq and hope to find a proper guildances, advices.
I am doing a Project from an online course so I can not give the material but I got my Project notebook on Github
I want to ask ...
- 3
1
vote
0
answers
35
views
Can back-bone of text-to-image GEN AI models utilised for classification?
With the advent of GEN AI (Stable Diffusion), we are able to create images with text. For eg. If i need to create a dog on beach during sunset; now in background this model needs to first get images ...
- 33
1
vote
1
answer
96
views
How was the token list determined for the tokenizer "cl100k_base"? [closed]
Does it have something to do with smoothing out the token frequencies to a desired distribution? If so, what's that distribution? And how is it achieved?
Is there a separate paper about it? Or should ...
- 23
0
votes
0
answers
46
views
What are the *non-cost-related* reasons RNN+Attention underperform Transformers?
There are obvious trainability and performance challenges with RNNs, such as having to process in serial and BPTT. But let's say we magically had an "optimal" set of weights for the RNN + ...
- 146
1
vote
2
answers
75
views
How is the padding mask incorporated in the attention formula?
I have been looking for the answer in other questions but no one tackled that. I want to ask you how is the padding mask considered in the formula of attention?
The attention formula taking into ...
- 563
0
votes
0
answers
103
views
Does anyone recognize this formula to quantify the likelihood that a transformer will generate the same response twice?
The idea is simple enough. Just multiply the likelihood of filling in the blank with the same result as the original response.
$$\prod_{s:substring}^{t:string}P(t|masked(t,s))$$
Motivation:
Rather ...
- 101
0
votes
1
answer
85
views
Fine-Tune Llama on main and auxiliary task
I am trying to fine-tune Llama model on two task at the same time, using hugging face library:
Main task: Causal language model like the model was initially trained for
A classification task based on ...
- 1
1
vote
0
answers
346
views
What is considered the pre-fill, and what is considered the decoding phase in this process?
I've seen conflicting information about this online so I'm looking for clarification. I'm dealing with the causal LLaMAF model specifically.
I used to think that a sequence of tokens is generated in, ...
- 11
0
votes
0
answers
56
views
A technique to show what tokens are relatively predicted by an LLM
I’m picturing a technique where you can see what an LLM is likely to respond with, which updates in real time.
It’s a bit trippy, but it’s like GitHub Copilot, in that there is predicted text while ...
- 123
0
votes
2
answers
62
views
Does fine-tuning a multilingual transformer model allow it to generalize to languages unseen in the fine-tuning dataset?
Example: https://huggingface.co/google/umt5-base
Note: UMT5 was only pre-trained on mC4 excluding any supervised training. Therefore, this model has to be fine-tuned before it is useable on a ...
- 103
1
vote
1
answer
50
views
What is an appropriate tool to use that takes in a large knowledge base in string form and can answer questions based on the knowledge base?
I have an issue where I'm trying to use the openAI API to input a very large custom knowledge base (exceeding 1GB) that allows the user to ask questions based on that base to receive intelligent ...
- 11
1
vote
1
answer
57
views
How does a multidimensional vector get fed into a single node in a neural network?
I mostly develop neural networks completely from scratch, like without libraries. I've been seeing, especially in NLP tasks, entire vectors, often representing words, get fed into a single node. I'm ...
- 11
3
votes
1
answer
107
views
How are sentences numerically encoded before passing them to neural networks?
I'm trying to understand NLP, how sentences can be used as input output in neural network architecture. As we know ANN is only compatible with number data. That's mean the sentences must be convert to ...
0
votes
1
answer
41
views
Sentence generation for limited vocabulary
I need to make a sentence generator for a limited set of vocabulary (about 600 words). The requirements are:
It must use only the words that are on the list, and never go beyond that;
It must produce ...
- 103
0
votes
0
answers
87
views
How to fine-tune a pre-trained model for customer service like tasks chatbot?
I want to make a trainable conversational bot that can respond to customer service questions. The bot should be able to adapt to different domains based on the dataset provided by the user (a company ...
2
votes
1
answer
273
views
What if we drop the causal mask in auto-regressive Transformer?
I understand the triangular causal mask in the attention is used to prevent tokens from "looking into the future", but why do we want to prevent that?
Let's suppose we have a model with ...
- 271
0
votes
0
answers
12
views
What ways are there to cluster an embedding space via binary branching?
Let’s say I have a corpus of text and generate embeddings for it (I’m new to this, so not too particular as to what type of embeddings to use).
I assume there is some function that can show me the top ...
- 123
0
votes
0
answers
21
views
Exploring NLP Techniques for Transforming Web Content into Engaging Q&A Formats
I'm looking for innovative methods or tools that leverage natural language processing (NLP) to transform web content into an interactive Q&A format. NLP has gotten so good that I no longer program ...
- 1
0
votes
2
answers
115
views
Why shouldn't the attention matrices $W^Q$, $W^K$, $W^V$ be the same?
My question is why the attention head matrices $W^Q$, $W^K$, $W^V$ should not be the same $W = W^Q =W^K= W^V$. In my understanding of transformer-based language models one attention head is ...
0
votes
0
answers
32
views
AI-driven tool for generating or finding short, context-aware jokes for online forum posts (GPT-4 not effective)
I've tried using GPT-4 to generate jokes with various prompts for my online forum posts, but most of the generated jokes were unfunny. For example, I asked my AI for a joke, and it said "your ...
- 1
1
vote
0
answers
93
views
How to work with multiple embeddings?
This is a conceptual gap that I have concerning embeddings, and would really appreciate some help closing it.
I understand when you embed a corpus for, let's say, a question-and-answer task you can ...
- 11
0
votes
0
answers
142
views
How big the context can be using HuggingFace models?
I'm new on AI, Neural Networks, ChatBots and all this ecosystem. I'm trying to use a classical example of pre-trained models, more specifically ...
- 101
3
votes
0
answers
92
views
Let's Verify Step by Step: Old wine in new bottles?
In their paper "Let's Verify Step by Step" OpenAI proudly presents a new way of reward learning which shall foster LLMs' capabilities of mathematical and logical reasoning:
We've trained a ...
0
votes
0
answers
9
views
How to compare word segmentation methods?
I am comparing a few methods of word segmentation in artificial language without dictionary and "golden" segmentation.
Let's say, idolikecats is splitted ...
- 1
0
votes
0
answers
54
views
NLP - Sentence reconstruction
I need to define a neural network for sentence reconstruction. Below the details:
The dataset is composed by a snapshot of wikipedia. We restricted the vocabolary to the 10K most frequent words, and ...
0
votes
0
answers
28
views
How to create dataset to extract information and classify intent using BERT?
Given a message: "Hey I am XYZ person (description about oneself), and I was thinking to launch a youtube video, wanted to get in touch with someone with similar experience", the model ...
0
votes
2
answers
69
views
If I freeze pre-trained model weights and than train a classifier on top of its embeddings does that called fine-tunning?
In the context of machine learning. If I freeze pre-trained model weights (for example, BERT) and then train a classifier on top of its embeddings, does that called fine-tuning?
0
votes
2
answers
356
views
How do I choose a good treshold for classification (using cosine similarity scores)?
I am using openai's text-embedding-ada-002 embeddings model to do a semantic search on a database of articles to find articles that are most related to a given ...
- 1
1
vote
0
answers
201
views
What is MLM & NSP loss function
Two objective functions are used during the BERT language
model pretraining step.
The first one is masked language
model (MLM) that randomly masks
15% of the
input tokens and the objective is to ...
- 121