Questions tagged [natural-language-processing]

For questions related to natural language processing (NLP), which is concerned with the interactions between computers and human (or natural) languages, in particular how to create programs that process and analyze large amounts of natural language data.

Filter by
Sorted by
Tagged with
0 votes
0 answers
17 views

ChatGPT and the P-NP Problem

According to ZDNet, it is an open question whether a transformer LLM like ChatGPT can facilitate the determination of a solution to the P-NP Problem. (See Can generative AI solve computer science's ...
J D's user avatar
  • 101
0 votes
1 answer
17 views

How to train a seq2seq model to rephrase input text following given rules

I want to train (fine-tune) a seq2seq model to perform the task of rephrasing input following these rules : 1- always follow the pattern "Entity Verb Entity" 2- only use simple sentences : ...
Wissem Boujlida's user avatar
0 votes
1 answer
26 views

How to get Llama-2 Rotary Embeddings?

I want to get the Llama-2 rotary embeddings. I do print(model) and get the following output: In the picture I highlight the rotary embeddings. How can get the ...
Christian01's user avatar
1 vote
1 answer
24 views

Data preparation for NLP model

I have data from our ticketing system. Currently using OpenNLP to create different models. For simplicity I have a 10k ticket's text as category final queue of the ticket. My questions: Is it ...
Milkmaid's user avatar
  • 111
0 votes
0 answers
6 views

Best prompt and model for fact-checking a text (disinformation/fake-news detection) [migrated]

Given a short text <text_to_check>, I want the LLM to check whether there are some facts stated in the text which are NOT true. So I want to detect 'disinformation' / 'fake news'. And the LLM ...
user2454869's user avatar
0 votes
1 answer
33 views

Why do current language models no longer generate to long or short texts?

One of the biggest strengths of ChatGPT is that it generates fitting text with respect to the input query. It usually stays on topic, anwers the question completely and especially does not start ...
Ricu's user avatar
  • 1
0 votes
0 answers
19 views

NLP for classifying a YES or NO response to a question

I'm currently working on a project that requires some feature extraction. The data I have is text and comes from an interview. The interviewer asks a question, the client responds, and the interviewer ...
altheconda's user avatar
1 vote
1 answer
40 views

AI: Inverse questions answering - quiz style: Verify descriptive answers to a static question

I'm exploring LLMs for educative purposes an came across the topic question answering, e. g. building a system that ingests documents like PDFs and is able to answer questions about its content. My ...
florian norbert bepunkt's user avatar
0 votes
1 answer
35 views

Keywords extractions from short names (table and column names) [closed]

I extract keywords that are table and column names (around 100,000 in the test). I process them in Python, and as a result, I get a CSV file with sample keywords: db_id type object_id keyword 1 a ...
tbo812's user avatar
  • 3
2 votes
0 answers
50 views

Why is an encoder + decoder model with L by L layers the same speed as as decoder only model with 2 L layers?

I was watching this lecture: https://youtu.be/27rNqGrTdSI?t=2295 In it the presenter stated that: "An encoder + decoder model with L by L layers is actually the same speed as as decoder only ...
shawn's user avatar
  • 21
0 votes
1 answer
65 views

Fine Tuning a Bert Transformer. How to label for emotions and train large scripts?

From what I have seen you can fine tune a Bert model to detect emotions by labelling single sentences. But if the text you want to evaluate is a large script with many sentences, do I need to split ...
arame3333's user avatar
0 votes
1 answer
21 views

How to label missing/default values for a named entity recognition dataset

I am building the training dataset for a named entity recognition model, with 2 tags: Name and Category and I am using a pre-...
mrang's user avatar
  • 3
0 votes
1 answer
32 views

How can BERT/Transformer models accept input batches of different sizes?

I understand that all inputs in a batch need to be of the same size. However, it seems BERT/Transformers models can accept batches with different sizes as input. How is that possible? I thought we ...
PS1's user avatar
  • 1
0 votes
1 answer
33 views

How word2vec de-embeds the special names in language models which output text

I am new to nlp field. I have some questions about word2vec embeddings. as I know they have a fixed size dictionary of vocabs. so definitely there some words which is not in that predefined dictionary ...
Farhang Amaji's user avatar
0 votes
1 answer
31 views

How does GPT like Decoder only conversational models distunguish the source of text?

In a conversational setting where two sources of text (user and the model) follow each other like below User: some text bla bla Model: another text bah bah User: bla bla bla Model: bah bah and so on, ...
meliksahturker's user avatar
0 votes
0 answers
28 views

Concatenation of Feature vectors in transformers before passing to fcnn

** As I am new to the field , the question might feel little abstract and naïve considering my experience. I am studying the Transformer architecture and trying to understand the various components ...
Buddha Dev Bhattacharjee's user avatar
0 votes
0 answers
19 views

Create samples out of documents for Causal Language Modelling

I want to create an input source for Causal Language model using Llama 2 model in hugging face. I have a set of documents which are scraped from a specific website and want to fine-tune on them. Each ...
Dimits's user avatar
  • 1
0 votes
0 answers
14 views

Has anyone tried to derive linguistic information from GPT internals?

In Stephen Wolfram's write up on the workings of GPT, he suggests that chat GPT may have identified invariant rules of human language that haven't been formalized yet, ie new linguistic findings, and ...
ak0000's user avatar
  • 195
0 votes
0 answers
20 views

Any research in "probe-tuning" of LLMs?

Is there any research in "probe-tuning" of LLMs, i.e., tuning LLM's parameter weights such that a specific probe (classifier) is more reliably detecting certain markers throughout the ...
leventov's user avatar
  • 101
0 votes
0 answers
14 views

What if in DPR (dense passage retrieval), the answer belongs to more than one passage?

In the DPR paper the dataset is expected to be in this format D = {<qi, pi+, pi,1-, ... >} With only one positive passage, but it is possible that the question requires an answer that spans ...
naren's user avatar
  • 1
0 votes
0 answers
170 views

Optimal Quantity of Training Data for Fine-Tuning an LLM: Is Bigger Always Better?

I am currently working on fine-tuning an LLM for a specific task, and I am trying to determine the optimal size for my training dataset. Intuitively, one might think that the more data, the better. ...
Peyman's user avatar
  • 544
0 votes
1 answer
13 views

Seq2Seq model- Confusing about the dimension of Seq2Seq model [closed]

I am new to Seq2Seq and hope to find a proper guildances, advices. I am doing a Project from an online course so I can not give the material but I got my Project notebook on Github I want to ask ...
QH.Chu's user avatar
  • 3
1 vote
0 answers
35 views

Can back-bone of text-to-image GEN AI models utilised for classification?

With the advent of GEN AI (Stable Diffusion), we are able to create images with text. For eg. If i need to create a dog on beach during sunset; now in background this model needs to first get images ...
prat__'s user avatar
  • 33
1 vote
1 answer
96 views

How was the token list determined for the tokenizer "cl100k_base"? [closed]

Does it have something to do with smoothing out the token frequencies to a desired distribution? If so, what's that distribution? And how is it achieved? Is there a separate paper about it? Or should ...
oliver.c's user avatar
0 votes
0 answers
46 views

What are the *non-cost-related* reasons RNN+Attention underperform Transformers?

There are obvious trainability and performance challenges with RNNs, such as having to process in serial and BPTT. But let's say we magically had an "optimal" set of weights for the RNN + ...
llllvvuu's user avatar
  • 146
1 vote
2 answers
75 views

How is the padding mask incorporated in the attention formula?

I have been looking for the answer in other questions but no one tackled that. I want to ask you how is the padding mask considered in the formula of attention? The attention formula taking into ...
Daviiid's user avatar
  • 563
0 votes
0 answers
103 views

Does anyone recognize this formula to quantify the likelihood that a transformer will generate the same response twice?

The idea is simple enough. Just multiply the likelihood of filling in the blank with the same result as the original response. $$\prod_{s:substring}^{t:string}P(t|masked(t,s))$$ Motivation: Rather ...
Andrew Johnson's user avatar
0 votes
1 answer
85 views

Fine-Tune Llama on main and auxiliary task

I am trying to fine-tune Llama model on two task at the same time, using hugging face library: Main task: Causal language model like the model was initially trained for A classification task based on ...
Dimits's user avatar
  • 1
1 vote
0 answers
346 views

What is considered the pre-fill, and what is considered the decoding phase in this process?

I've seen conflicting information about this online so I'm looking for clarification. I'm dealing with the causal LLaMAF model specifically. I used to think that a sequence of tokens is generated in, ...
jgeddes's user avatar
  • 11
0 votes
0 answers
56 views

A technique to show what tokens are relatively predicted by an LLM

I’m picturing a technique where you can see what an LLM is likely to respond with, which updates in real time. It’s a bit trippy, but it’s like GitHub Copilot, in that there is predicted text while ...
hmltn's user avatar
  • 123
0 votes
2 answers
62 views

Does fine-tuning a multilingual transformer model allow it to generalize to languages unseen in the fine-tuning dataset?

Example: https://huggingface.co/google/umt5-base Note: UMT5 was only pre-trained on mC4 excluding any supervised training. Therefore, this model has to be fine-tuned before it is useable on a ...
Michał B.'s user avatar
1 vote
1 answer
50 views

What is an appropriate tool to use that takes in a large knowledge base in string form and can answer questions based on the knowledge base?

I have an issue where I'm trying to use the openAI API to input a very large custom knowledge base (exceeding 1GB) that allows the user to ask questions based on that base to receive intelligent ...
Hou Wan's user avatar
  • 11
1 vote
1 answer
57 views

How does a multidimensional vector get fed into a single node in a neural network?

I mostly develop neural networks completely from scratch, like without libraries. I've been seeing, especially in NLP tasks, entire vectors, often representing words, get fed into a single node. I'm ...
Jake StBu's user avatar
3 votes
1 answer
107 views

How are sentences numerically encoded before passing them to neural networks?

I'm trying to understand NLP, how sentences can be used as input output in neural network architecture. As we know ANN is only compatible with number data. That's mean the sentences must be convert to ...
Muhammad Ikhwan Perwira's user avatar
0 votes
1 answer
41 views

Sentence generation for limited vocabulary

I need to make a sentence generator for a limited set of vocabulary (about 600 words). The requirements are: It must use only the words that are on the list, and never go beyond that; It must produce ...
Slavus's user avatar
  • 103
0 votes
0 answers
87 views

How to fine-tune a pre-trained model for customer service like tasks chatbot?

I want to make a trainable conversational bot that can respond to customer service questions. The bot should be able to adapt to different domains based on the dataset provided by the user (a company ...
Mohamed saad's user avatar
2 votes
1 answer
273 views

What if we drop the causal mask in auto-regressive Transformer?

I understand the triangular causal mask in the attention is used to prevent tokens from "looking into the future", but why do we want to prevent that? Let's suppose we have a model with ...
nalzok's user avatar
  • 271
0 votes
0 answers
12 views

What ways are there to cluster an embedding space via binary branching?

Let’s say I have a corpus of text and generate embeddings for it (I’m new to this, so not too particular as to what type of embeddings to use). I assume there is some function that can show me the top ...
hmltn's user avatar
  • 123
0 votes
0 answers
21 views

Exploring NLP Techniques for Transforming Web Content into Engaging Q&A Formats

I'm looking for innovative methods or tools that leverage natural language processing (NLP) to transform web content into an interactive Q&A format. NLP has gotten so good that I no longer program ...
8ta4's user avatar
  • 1
0 votes
2 answers
115 views

Why shouldn't the attention matrices $W^Q$, $W^K$, $W^V$ be the same?

My question is why the attention head matrices $W^Q$, $W^K$, $W^V$ should not be the same $W = W^Q =W^K= W^V$. In my understanding of transformer-based language models one attention head is ...
Hans-Peter Stricker's user avatar
0 votes
0 answers
32 views

AI-driven tool for generating or finding short, context-aware jokes for online forum posts (GPT-4 not effective)

I've tried using GPT-4 to generate jokes with various prompts for my online forum posts, but most of the generated jokes were unfunny. For example, I asked my AI for a joke, and it said "your ...
8ta4's user avatar
  • 1
1 vote
0 answers
93 views

How to work with multiple embeddings?

This is a conceptual gap that I have concerning embeddings, and would really appreciate some help closing it. I understand when you embed a corpus for, let's say, a question-and-answer task you can ...
Ian Murray's user avatar
0 votes
0 answers
142 views

How big the context can be using HuggingFace models?

I'm new on AI, Neural Networks, ChatBots and all this ecosystem. I'm trying to use a classical example of pre-trained models, more specifically ...
Magno C's user avatar
  • 101
3 votes
0 answers
92 views

Let's Verify Step by Step: Old wine in new bottles?

In their paper "Let's Verify Step by Step" OpenAI proudly presents a new way of reward learning which shall foster LLMs' capabilities of mathematical and logical reasoning: We've trained a ...
Hans-Peter Stricker's user avatar
0 votes
0 answers
9 views

How to compare word segmentation methods?

I am comparing a few methods of word segmentation in artificial language without dictionary and "golden" segmentation. Let's say, idolikecats is splitted ...
dobrowol's user avatar
0 votes
0 answers
54 views

NLP - Sentence reconstruction

I need to define a neural network for sentence reconstruction. Below the details: The dataset is composed by a snapshot of wikipedia. We restricted the vocabolary to the 10K most frequent words, and ...
jacopodabramo's user avatar
0 votes
0 answers
28 views

How to create dataset to extract information and classify intent using BERT?

Given a message: "Hey I am XYZ person (description about oneself), and I was thinking to launch a youtube video, wanted to get in touch with someone with similar experience", the model ...
thecalendar's user avatar
0 votes
2 answers
69 views

If I freeze pre-trained model weights and than train a classifier on top of its embeddings does that called fine-tunning?

In the context of machine learning. If I freeze pre-trained model weights (for example, BERT) and then train a classifier on top of its embeddings, does that called fine-tuning?
Lampent96's user avatar
0 votes
2 answers
356 views

How do I choose a good treshold for classification (using cosine similarity scores)?

I am using openai's text-embedding-ada-002 embeddings model to do a semantic search on a database of articles to find articles that are most related to a given ...
Stefan's user avatar
  • 1
1 vote
0 answers
201 views

What is MLM & NSP loss function

Two objective functions are used during the BERT language model pretraining step. The first one is masked language model (MLM) that randomly masks 15% of the input tokens and the objective is to ...
XYZ's user avatar
  • 121

1
2 3 4 5
15