Questions tagged [natural-language-processing]

For questions related to natural language processing (NLP), which is concerned with the interactions between computers and human (or natural) languages, in particular how to create programs that process and analyze large amounts of natural language data.

Filter by
Sorted by
Tagged with
0 votes
0 answers
8 views

extractive and generative q&a systems

I would like to build a q&a system because of the advancements in generative ai especially large language models (LLM) aka chatgpt and co. it seems like everyone wants to build this. So I started ...
Khan's user avatar
  • 175
0 votes
0 answers
12 views

Is Training Separate NER Models for Specific Labels a Viable Strategy for Improved Accuracy?

In my project, I'm utilizing Named Entity Recognition (NER) to identify crucial variables in prompts categorized into specific domains. However, incorporating numerous labels in the training data has ...
Raghul Azhagaiah's user avatar
0 votes
0 answers
22 views

Prefix tuning in LLM uses learnable vectors to fine tune the model

I would like to implement a new architecture for Transformer. Below description is my thought. Prefix tuning in LLM uses learnable vectors to fine tune the model. Is there a way to use the output ...
jackson's user avatar
1 vote
3 answers
62 views

Would maximizing (instead of minimizing) error of an LLM/HMM lead to complex behavior?

Imagine we have some sort of "next token predictor," either with transformer architecture, LSTM, or just a HMM (though the terminology I use here will be less aligned to HMMs, I believe the ...
BigMistake's user avatar
1 vote
1 answer
45 views

NLP "small" model to improve "big" model

When training the model for NLP is it important to get rid of data which has "bad semantic" for learning process? My plan is to create a "small model" which can decide whether data ...
Milkmaid's user avatar
  • 125
1 vote
0 answers
20 views

What is the input to an encoder-decoder transformer in next word prediction task?

I'm trying to understand how encoder-decoder architectures are used, or if they are used at all, for generative tasks that do not require an explicit prompt (ie. machine translation, summarization, ...
mehsheenman's user avatar
1 vote
1 answer
88 views

Masking in Decoder of Transformer

I understand that the masked multi-head attention block ensures that generation of token at time step t doesn't rely on subsequent tokens of the input. But the residual connection which adds the input ...
SAGALPREET SINGH's user avatar
0 votes
1 answer
33 views

If the unigram precision is (N-1)/N, then the bigram precision is :

Consider the following machine translation scenario. The reference translation has N words (do not consider sentence beginner ‘hat’ and sentence finisher ‘dot’). The machine output also has N words. ...
Geeklovenerds's user avatar
0 votes
1 answer
37 views

what are the applications scenarios for prefix decoder LMs

Motivated by this post wherein one of the comments mentioned the use-case for encoder-decoder LM. I wanted to know when to use prefix-decoder LM? vis a vis encoder-decoder or causal decoder only ...
Singh's user avatar
  • 1
0 votes
0 answers
23 views

How to measure similarities between text (word to word or word to phrase)?

Is there a way to measure to measure similarities between two pieces of text ? Think about the case where you have an image captioning model but you only want to deal/use specific class names. e.g. ...
Souhaielrmx's user avatar
0 votes
2 answers
70 views

Why encoders are required in Transformers

In the original Transformers paper why encoder is added when a decoder alone can do what an encoder can do (like multi-head attention, feed-forward NN etc....). I mean even a decoder also has the same ...
Swastik's user avatar
0 votes
1 answer
36 views

Setting number of rows returned by vector stores

When using vector stores like pinecone or Faiss from langchain, is it possible to set the number of records returned based on similarity search? For example, consider the following code, is there a ...
Karl 17302's user avatar
0 votes
0 answers
18 views

Exploring the Similarity of Sibling’s Voices Using Automatic Speaker Recognition

I want to start project on Exploring the Similarity of Sibling’s Voices Using Automatic Speaker Recognition Everyone has a unique voice, because of the different structure of their articulatory ...
Alan Turing's user avatar
1 vote
0 answers
67 views

LLM for Postgres

I have a postgres database with 200+ Tables. Each table contains information about my supply inventory. It also contains columns which are JSON and there are nested JSON as well. There are ...
Shivkumar Mallesappa's user avatar
2 votes
1 answer
120 views

Aren't context lengths for transformers an artificial restriction?

Let's focus on the case of decoder-only transformers, where I am using algorithm 10 from "Formal Algorithms for Transformers" by Mary Phung and Marcus Hutter as a reference. : https://i....
Robert Wegner's user avatar
1 vote
0 answers
23 views

Is there any well-established work that allows robots to communicate their decision-making using natural language?

I am searching for a well-established work that allows robots to communicate their decision-making using natural language. For example, a robot's explanation could be "I did [task1] because [...
jigz's user avatar
  • 21
0 votes
0 answers
104 views

Is ChatGPT a viable strategy for solving the P-NP Problem?

According to ZDNet, it is an open question whether a transformer LLM like ChatGPT can facilitate the determination of a solution to the P-NP Problem. (See Can generative AI solve computer science's ...
J D's user avatar
  • 101
0 votes
1 answer
35 views

How to train a seq2seq model to rephrase input text following given rules

I want to train (fine-tune) a seq2seq model to perform the task of rephrasing input following these rules : 1- always follow the pattern "Entity Verb Entity" 2- only use simple sentences : ...
Wissem Boujlida's user avatar
-1 votes
1 answer
233 views

How to get Llama-2 Rotary Embeddings?

I want to get the Llama-2 rotary embeddings. I do print(model) and get the following output: In the picture I highlight the rotary embeddings. How can get the ...
Christian01's user avatar
1 vote
2 answers
48 views

Data preparation for NLP model

I have data from our ticketing system. Currently using OpenNLP to create different models. For simplicity I have a 10k ticket's text as category final queue of the ticket. My questions: Is it ...
Milkmaid's user avatar
  • 125
0 votes
1 answer
48 views

Why do current language models no longer generate to long or short texts?

One of the biggest strengths of ChatGPT is that it generates fitting text with respect to the input query. It usually stays on topic, anwers the question completely and especially does not start ...
Ricu's user avatar
  • 1
0 votes
0 answers
49 views

NLP for classifying a YES or NO response to a question

I'm currently working on a project that requires some feature extraction. The data I have is text and comes from an interview. The interviewer asks a question, the client responds, and the interviewer ...
altheconda's user avatar
1 vote
1 answer
51 views

AI: Inverse questions answering - quiz style: Verify descriptive answers to a static question

I'm exploring LLMs for educative purposes an came across the topic question answering, e. g. building a system that ingests documents like PDFs and is able to answer questions about its content. My ...
florian norbert bepunkt's user avatar
0 votes
1 answer
38 views

Keywords extractions from short names (table and column names) [closed]

I extract keywords that are table and column names (around 100,000 in the test). I process them in Python, and as a result, I get a CSV file with sample keywords: db_id type object_id keyword 1 a ...
tbo812's user avatar
  • 3
2 votes
0 answers
61 views

Why is an encoder + decoder model with L by L layers the same speed as as decoder only model with 2 L layers?

I was watching this lecture: https://youtu.be/27rNqGrTdSI?t=2295 In it the presenter stated that: "An encoder + decoder model with L by L layers is actually the same speed as as decoder only ...
shawn's user avatar
  • 21
0 votes
1 answer
92 views

Fine Tuning a Bert Transformer. How to label for emotions and train large scripts?

From what I have seen you can fine tune a Bert model to detect emotions by labelling single sentences. But if the text you want to evaluate is a large script with many sentences, do I need to split ...
arame3333's user avatar
0 votes
1 answer
27 views

How to label missing/default values for a named entity recognition dataset

I am building the training dataset for a named entity recognition model, with 2 tags: Name and Category and I am using a pre-...
mrang's user avatar
  • 3
0 votes
1 answer
53 views

How can BERT/Transformer models accept input batches of different sizes?

I understand that all inputs in a batch need to be of the same size. However, it seems BERT/Transformers models can accept batches with different sizes as input. How is that possible? I thought we ...
PS1's user avatar
  • 101
0 votes
1 answer
40 views

How word2vec de-embeds the special names in language models which output text

I am new to nlp field. I have some questions about word2vec embeddings. as I know they have a fixed size dictionary of vocabs. so definitely there some words which is not in that predefined dictionary ...
Farhang Amaji's user avatar
0 votes
1 answer
41 views

How does GPT like Decoder only conversational models distunguish the source of text?

In a conversational setting where two sources of text (user and the model) follow each other like below User: some text bla bla Model: another text bah bah User: bla bla bla Model: bah bah and so on, ...
meliksahturker's user avatar
1 vote
0 answers
73 views

Concatenation of Feature vectors in transformers before passing to fcnn

** As I am new to the field , the question might feel little abstract and naïve considering my experience. I am studying the Transformer architecture and trying to understand the various components ...
Buddha Dev Bhattacharjee's user avatar
0 votes
0 answers
29 views

Create samples out of documents for Causal Language Modelling

I want to create an input source for Causal Language model using Llama 2 model in hugging face. I have a set of documents which are scraped from a specific website and want to fine-tune on them. Each ...
Dimits's user avatar
  • 1
0 votes
0 answers
16 views

Has anyone tried to derive linguistic information from GPT internals?

In Stephen Wolfram's write up on the workings of GPT, he suggests that chat GPT may have identified invariant rules of human language that haven't been formalized yet, ie new linguistic findings, and ...
ak0000's user avatar
  • 205
0 votes
0 answers
22 views

Any research in "probe-tuning" of LLMs?

Is there any research in "probe-tuning" of LLMs, i.e., tuning LLM's parameter weights such that a specific probe (classifier) is more reliably detecting certain markers throughout the ...
leventov's user avatar
  • 101
0 votes
0 answers
20 views

What if in DPR (dense passage retrieval), the answer belongs to more than one passage?

In the DPR paper the dataset is expected to be in this format D = {<qi, pi+, pi,1-, ... >} With only one positive passage, but it is possible that the question requires an answer that spans ...
naren's user avatar
  • 1
0 votes
0 answers
249 views

Optimal Quantity of Training Data for Fine-Tuning an LLM: Is Bigger Always Better?

I am currently working on fine-tuning an LLM for a specific task, and I am trying to determine the optimal size for my training dataset. Intuitively, one might think that the more data, the better. ...
Peyman's user avatar
  • 554
0 votes
1 answer
21 views

Seq2Seq model- Confusing about the dimension of Seq2Seq model [closed]

I am new to Seq2Seq and hope to find a proper guildances, advices. I am doing a Project from an online course so I can not give the material but I got my Project notebook on Github I want to ask ...
QH.Chu's user avatar
  • 3
1 vote
0 answers
37 views

Can back-bone of text-to-image GEN AI models utilised for classification?

With the advent of GEN AI (Stable Diffusion), we are able to create images with text. For eg. If i need to create a dog on beach during sunset; now in background this model needs to first get images ...
prat__'s user avatar
  • 33
1 vote
1 answer
180 views

How was the token list determined for the tokenizer "cl100k_base"? [closed]

Does it have something to do with smoothing out the token frequencies to a desired distribution? If so, what's that distribution? And how is it achieved? Is there a separate paper about it? Or should ...
oliver.c's user avatar
0 votes
0 answers
47 views

What are the *non-cost-related* reasons RNN+Attention underperform Transformers?

There are obvious trainability and performance challenges with RNNs, such as having to process in serial and BPTT. But let's say we magically had an "optimal" set of weights for the RNN + ...
llllvvuu's user avatar
  • 146
1 vote
2 answers
136 views

How is the padding mask incorporated in the attention formula?

I have been looking for the answer in other questions but no one tackled that. I want to ask you how is the padding mask considered in the formula of attention? The attention formula taking into ...
Daviiid's user avatar
  • 573
0 votes
0 answers
104 views

Does anyone recognize this formula to quantify the likelihood that a transformer will generate the same response twice?

The idea is simple enough. Just multiply the likelihood of filling in the blank with the same result as the original response. $$\prod_{s:substring}^{t:string}P(t|masked(t,s))$$ Motivation: Rather ...
Andrew Johnson's user avatar
0 votes
1 answer
120 views

Fine-Tune Llama on main and auxiliary task

I am trying to fine-tune Llama model on two task at the same time, using hugging face library: Main task: Causal language model like the model was initially trained for A classification task based on ...
Dimits's user avatar
  • 1
1 vote
1 answer
686 views

What is considered the pre-fill, and what is considered the decoding phase in this process?

I've seen conflicting information about this online so I'm looking for clarification. I'm dealing with the causal LLaMAF model specifically. I used to think that a sequence of tokens is generated in, ...
jgeddes's user avatar
  • 11
0 votes
0 answers
103 views

A technique to show what tokens are relatively predicted by an LLM

I’m picturing a technique where you can see what an LLM is likely to respond with, which updates in real time. It’s a bit trippy, but it’s like GitHub Copilot, in that there is predicted text while ...
Julius H.'s user avatar
  • 133
0 votes
2 answers
85 views

Does fine-tuning a multilingual transformer model allow it to generalize to languages unseen in the fine-tuning dataset?

Example: https://huggingface.co/google/umt5-base Note: UMT5 was only pre-trained on mC4 excluding any supervised training. Therefore, this model has to be fine-tuned before it is useable on a ...
Michał B.'s user avatar
1 vote
1 answer
55 views

What is an appropriate tool to use that takes in a large knowledge base in string form and can answer questions based on the knowledge base?

I have an issue where I'm trying to use the openAI API to input a very large custom knowledge base (exceeding 1GB) that allows the user to ask questions based on that base to receive intelligent ...
Hou Wan's user avatar
  • 11
1 vote
1 answer
66 views

How does a multidimensional vector get fed into a single node in a neural network?

I mostly develop neural networks completely from scratch, like without libraries. I've been seeing, especially in NLP tasks, entire vectors, often representing words, get fed into a single node. I'm ...
Jake StBu's user avatar
3 votes
1 answer
137 views

How are sentences numerically encoded before passing them to neural networks?

I'm trying to understand NLP, how sentences can be used as input output in neural network architecture. As we know ANN is only compatible with number data. That's mean the sentences must be convert to ...
Muhammad Ikhwan Perwira's user avatar
0 votes
1 answer
69 views

Sentence generation for limited vocabulary

I need to make a sentence generator for a limited set of vocabulary (about 600 words). The requirements are: It must use only the words that are on the list, and never go beyond that; It must produce ...
Slavus's user avatar
  • 103

1
2 3 4 5
15