Questions tagged [natural-language-processing]
For questions related to natural language processing (NLP), which is concerned with the interactions between computers and human (or natural) languages, in particular how to create programs that process and analyze large amounts of natural language data.
738
questions
0
votes
0
answers
9
views
Prefix tuning in LLM uses learnable vectors to fine tune the model
I would like to implement a new architecture for Transformer.
Below description is my thought.
Prefix tuning in LLM uses learnable vectors to fine tune the model.
Is there a way to use the output ...
- 1
1
vote
3
answers
62
views
Would maximizing (instead of minimizing) error of an LLM/HMM lead to complex behavior?
Imagine we have some sort of "next token predictor," either with transformer architecture, LSTM, or just a HMM (though the terminology I use here will be less aligned to HMMs, I believe the ...
- 113
1
vote
1
answer
44
views
NLP "small" model to improve "big" model
When training the model for NLP is it important to get rid of data which has "bad semantic" for learning process?
My plan is to create a "small model" which can decide whether data ...
- 125
1
vote
0
answers
19
views
What is the input to an encoder-decoder transformer in next word prediction task?
I'm trying to understand how encoder-decoder architectures are used, or if they are used at all, for generative tasks that do not require an explicit prompt (ie. machine translation, summarization, ...
- 111
1
vote
1
answer
87
views
Masking in Decoder of Transformer
I understand that the masked multi-head attention block ensures that generation of token at time step t doesn't rely on subsequent tokens of the input. But the residual connection which adds the input ...
- 107
0
votes
1
answer
32
views
If the unigram precision is (N-1)/N, then the bigram precision is :
Consider the following machine translation scenario. The reference translation has N words (do not consider sentence beginner ‘hat’ and sentence finisher ‘dot’). The machine output also has N words. ...
- 101
0
votes
1
answer
35
views
what are the applications scenarios for prefix decoder LMs
Motivated by this post wherein one of the comments mentioned the use-case for encoder-decoder LM. I wanted to know when to use prefix-decoder LM? vis a vis encoder-decoder or causal decoder only ...
- 1
0
votes
0
answers
23
views
How to measure similarities between text (word to word or word to phrase)?
Is there a way to measure to measure similarities between two pieces of text ? Think about the case where you have an image captioning model but you only want to deal/use specific class names. e.g. ...
- 31
0
votes
0
answers
22
views
Deep learning to identify and classify first and last names into gender, age range and probable region of origin [closed]
I have a bit of a question as to how I could architect a pipeline of deep learning for the following use case:
gender identification from a full name (joseph is male, josephine is female, sasha is ...
- 1
0
votes
2
answers
69
views
Why encoders are required in Transformers
In the original Transformers paper why encoder is added when a decoder alone can do what an encoder can do (like multi-head attention, feed-forward NN etc....).
I mean even a decoder also has the same ...
- 1
0
votes
1
answer
32
views
Setting number of rows returned by vector stores
When using vector stores like pinecone or Faiss from langchain, is it possible to set the number of records returned based on similarity search?
For example, consider the following code, is there a ...
0
votes
0
answers
18
views
Exploring the Similarity of Sibling’s Voices Using Automatic Speaker Recognition
I want to start project on Exploring the Similarity of Sibling’s Voices Using Automatic Speaker Recognition
Everyone has a unique voice, because of the different structure of their articulatory ...
1
vote
0
answers
59
views
LLM for Postgres
I have a postgres database with 200+ Tables. Each table contains information about my supply inventory. It also contains columns which are JSON and there are nested JSON as well. There are ...
2
votes
1
answer
111
views
Aren't context lengths for transformers an artificial restriction?
Let's focus on the case of decoder-only transformers, where I am using algorithm 10 from "Formal Algorithms for Transformers" by Mary Phung and Marcus Hutter as a reference.
: https://i....
1
vote
0
answers
23
views
Is there any well-established work that allows robots to communicate their decision-making using natural language?
I am searching for a well-established work that allows robots to communicate their decision-making using natural language.
For example, a robot's explanation could be "I did [task1] because [...
- 21
0
votes
0
answers
97
views
Is ChatGPT a viable strategy for solving the P-NP Problem?
According to ZDNet, it is an open question whether a transformer LLM like ChatGPT can facilitate the determination of a solution to the P-NP Problem. (See Can generative AI solve computer science's ...
- 101
0
votes
1
answer
35
views
How to train a seq2seq model to rephrase input text following given rules
I want to train (fine-tune) a seq2seq model to perform the task of rephrasing input following these rules :
1- always follow the pattern "Entity Verb Entity"
2- only use simple sentences : ...
-1
votes
1
answer
216
views
How to get Llama-2 Rotary Embeddings?
I want to get the Llama-2 rotary embeddings. I do print(model) and get the following output:
In the picture I highlight the rotary embeddings.
How can get the ...
- 99
1
vote
2
answers
48
views
Data preparation for NLP model
I have data from our ticketing system. Currently using OpenNLP to create different models.
For simplicity I have a 10k ticket's text as category final queue of the ticket.
My questions:
Is it ...
- 125
0
votes
1
answer
47
views
Why do current language models no longer generate to long or short texts?
One of the biggest strengths of ChatGPT is that it generates fitting text with respect to the input query. It usually stays on topic, anwers the question completely and especially does not start ...
- 1
0
votes
0
answers
45
views
NLP for classifying a YES or NO response to a question
I'm currently working on a project that requires some feature extraction. The data I have is text and comes from an interview. The interviewer asks a question, the client responds, and the interviewer ...
- 101
1
vote
1
answer
51
views
AI: Inverse questions answering - quiz style: Verify descriptive answers to a static question
I'm exploring LLMs for educative purposes an came across the topic question answering, e. g. building a system that ingests documents like PDFs and is able to answer questions about its content.
My ...
0
votes
1
answer
38
views
Keywords extractions from short names (table and column names) [closed]
I extract keywords that are table and column names (around 100,000 in the test).
I process them in Python, and as a result, I get a CSV file with sample keywords:
db_id
type
object_id
keyword
1
a
...
- 3
2
votes
0
answers
61
views
Why is an encoder + decoder model with L by L layers the same speed as as decoder only model with 2 L layers?
I was watching this lecture: https://youtu.be/27rNqGrTdSI?t=2295
In it the presenter stated that:
"An encoder + decoder model with L by L layers is actually the same speed as as decoder only ...
- 21
0
votes
1
answer
90
views
Fine Tuning a Bert Transformer. How to label for emotions and train large scripts?
From what I have seen you can fine tune a Bert model to detect emotions by labelling single sentences.
But if the text you want to evaluate is a large script with many sentences, do I need to split ...
- 53
0
votes
1
answer
27
views
How to label missing/default values for a named entity recognition dataset
I am building the training dataset for a named entity recognition model, with 2 tags: Name and Category and I am using a pre-...
- 3
0
votes
1
answer
46
views
How can BERT/Transformer models accept input batches of different sizes?
I understand that all inputs in a batch need to be of the same size. However, it seems BERT/Transformers models can accept batches with different sizes as input.
How is that possible? I thought we ...
- 101
0
votes
1
answer
40
views
How word2vec de-embeds the special names in language models which output text
I am new to nlp field. I have some questions about word2vec embeddings. as I know they have a fixed size dictionary of vocabs. so definitely there some words which is not in that predefined dictionary ...
- 101
0
votes
1
answer
41
views
How does GPT like Decoder only conversational models distunguish the source of text?
In a conversational setting where two sources of text (user and the model) follow each other like below
User: some text bla bla
Model: another text bah bah
User: bla bla bla
Model: bah bah
and so on, ...
- 296
1
vote
0
answers
69
views
Concatenation of Feature vectors in transformers before passing to fcnn
** As I am new to the field , the question might feel little abstract and naïve considering my experience.
I am studying the Transformer architecture and trying to understand the various components ...
0
votes
0
answers
29
views
Create samples out of documents for Causal Language Modelling
I want to create an input source for Causal Language model using Llama 2 model in hugging face. I have a set of documents which are scraped from a specific website and want to fine-tune on them. Each ...
- 1
0
votes
0
answers
16
views
Has anyone tried to derive linguistic information from GPT internals?
In Stephen Wolfram's write up on the workings of GPT, he suggests that chat GPT may have identified invariant rules of human language that haven't been formalized yet, ie new linguistic findings, and ...
- 205
0
votes
0
answers
22
views
Any research in "probe-tuning" of LLMs?
Is there any research in "probe-tuning" of LLMs, i.e., tuning LLM's parameter weights such that a specific probe (classifier) is more reliably detecting certain markers throughout the ...
- 101
0
votes
0
answers
18
views
What if in DPR (dense passage retrieval), the answer belongs to more than one passage?
In the DPR paper
the dataset is expected to be in this format D = {<qi, pi+, pi,1-, ... >}
With only one positive passage, but it is possible that the question requires an answer that spans ...
- 1
0
votes
0
answers
246
views
Optimal Quantity of Training Data for Fine-Tuning an LLM: Is Bigger Always Better?
I am currently working on fine-tuning an LLM for a specific task, and I am trying to determine the optimal size for my training dataset. Intuitively, one might think that the more data, the better. ...
- 554
0
votes
1
answer
21
views
Seq2Seq model- Confusing about the dimension of Seq2Seq model [closed]
I am new to Seq2Seq and hope to find a proper guildances, advices.
I am doing a Project from an online course so I can not give the material but I got my Project notebook on Github
I want to ask ...
- 3
1
vote
0
answers
37
views
Can back-bone of text-to-image GEN AI models utilised for classification?
With the advent of GEN AI (Stable Diffusion), we are able to create images with text. For eg. If i need to create a dog on beach during sunset; now in background this model needs to first get images ...
- 33
1
vote
1
answer
174
views
How was the token list determined for the tokenizer "cl100k_base"? [closed]
Does it have something to do with smoothing out the token frequencies to a desired distribution? If so, what's that distribution? And how is it achieved?
Is there a separate paper about it? Or should ...
- 35
0
votes
0
answers
47
views
What are the *non-cost-related* reasons RNN+Attention underperform Transformers?
There are obvious trainability and performance challenges with RNNs, such as having to process in serial and BPTT. But let's say we magically had an "optimal" set of weights for the RNN + ...
- 146
1
vote
2
answers
127
views
How is the padding mask incorporated in the attention formula?
I have been looking for the answer in other questions but no one tackled that. I want to ask you how is the padding mask considered in the formula of attention?
The attention formula taking into ...
- 573
0
votes
0
answers
103
views
Does anyone recognize this formula to quantify the likelihood that a transformer will generate the same response twice?
The idea is simple enough. Just multiply the likelihood of filling in the blank with the same result as the original response.
$$\prod_{s:substring}^{t:string}P(t|masked(t,s))$$
Motivation:
Rather ...
- 101
0
votes
1
answer
117
views
Fine-Tune Llama on main and auxiliary task
I am trying to fine-tune Llama model on two task at the same time, using hugging face library:
Main task: Causal language model like the model was initially trained for
A classification task based on ...
- 1
1
vote
1
answer
656
views
What is considered the pre-fill, and what is considered the decoding phase in this process?
I've seen conflicting information about this online so I'm looking for clarification. I'm dealing with the causal LLaMAF model specifically.
I used to think that a sequence of tokens is generated in, ...
- 11
0
votes
0
answers
92
views
A technique to show what tokens are relatively predicted by an LLM
I’m picturing a technique where you can see what an LLM is likely to respond with, which updates in real time.
It’s a bit trippy, but it’s like GitHub Copilot, in that there is predicted text while ...
- 133
0
votes
2
answers
83
views
Does fine-tuning a multilingual transformer model allow it to generalize to languages unseen in the fine-tuning dataset?
Example: https://huggingface.co/google/umt5-base
Note: UMT5 was only pre-trained on mC4 excluding any supervised training. Therefore, this model has to be fine-tuned before it is useable on a ...
- 103
1
vote
1
answer
55
views
What is an appropriate tool to use that takes in a large knowledge base in string form and can answer questions based on the knowledge base?
I have an issue where I'm trying to use the openAI API to input a very large custom knowledge base (exceeding 1GB) that allows the user to ask questions based on that base to receive intelligent ...
- 11
1
vote
1
answer
65
views
How does a multidimensional vector get fed into a single node in a neural network?
I mostly develop neural networks completely from scratch, like without libraries. I've been seeing, especially in NLP tasks, entire vectors, often representing words, get fed into a single node. I'm ...
- 11
3
votes
1
answer
135
views
How are sentences numerically encoded before passing them to neural networks?
I'm trying to understand NLP, how sentences can be used as input output in neural network architecture. As we know ANN is only compatible with number data. That's mean the sentences must be convert to ...
0
votes
1
answer
66
views
Sentence generation for limited vocabulary
I need to make a sentence generator for a limited set of vocabulary (about 600 words). The requirements are:
It must use only the words that are on the list, and never go beyond that;
It must produce ...
- 103
0
votes
0
answers
118
views
How to fine-tune a pre-trained model for customer service like tasks chatbot?
I want to make a trainable conversational bot that can respond to customer service questions. The bot should be able to adapt to different domains based on the dataset provided by the user (a company ...
