Questions tagged [natural-language-processing]

For questions related to natural language processing (NLP), which is concerned with the interactions between computers and human (or natural) languages, in particular how to create programs that process and analyze large amounts of natural language data.

Filter by
Sorted by
Tagged with
-1 votes
0 answers
11 views

What is the best language model for summarising legal documents? [closed]

I have a set of >20,000 legal documents that I want to summarise key details of. For example, parties, transaction object, transaction price, geographic scope, etc. Currently I'm using the OpenAI ...
Matthias's user avatar
1 vote
2 answers
58 views

What technique is used for training Large Language Models like GPT?

I'm learning about GenAI, such as GPT (Generative Pretrained Transformer), and I'm particularly interested in understanding the training techniques used for these models. Deep learning, generally, can ...
Exploring's user avatar
  • 319
1 vote
1 answer
54 views

Why do Transformer decoders use masked self attention when producing new tokens?

I've been reading that transformer decoders use masked self attention so that the decoder can't cheat by looking ahead. For example, when predicting the 6th token in the sequence we shouldn't have ...
Kiran Manicka's user avatar
0 votes
0 answers
25 views

How using sinusoidal encoding for positional encoding saves space in comparison to binary encoding? [closed]

Context: The blog - https://kazemnejad.com/blog/transformer_architecture_positional_encoding/ mentions that using binary encoding would waste the space. Using the ...
Deepak Tatyaji Ahire's user avatar
0 votes
0 answers
11 views

Using a HMM for grammar checking

I want to try to make an HMM from scratch for POS tagging, which I would extend to grammar checking. I understand there's much better ways for a grammar checker but this is just as a learning ...
skarokin's user avatar
0 votes
1 answer
33 views

What language model can convert normal text to JSON data

I have tried training T5-small, T5-base and T5-Large on around 15K rows of data where input data was something like but I did not get desired results ...
Arjun Singh's user avatar
0 votes
0 answers
22 views

Where I can find SOTA models of information retrieval?

Can someone tell me where I can find SOTA models of information retrieval? My task is to rank documents by given query by semantic search of embedding. I know that models like ColBERT, SPLADE solve ...
prostak's user avatar
  • 103
0 votes
0 answers
24 views

Why won't ChatOllama work with GraphCypherQAChain?

I am trying to get a QA Chain that uses mistral to work, but the Chain fails to restrict the LLM to solely use the relationships in the Knowledge Graph (in Neo4j). The code is: ...
Índio's user avatar
  • 101
0 votes
0 answers
16 views

Does the fixed context in attention mechanism is accquired after getting the decoder hidden layer of the first hidden state?

here, the fixed context vector (ci) is used for the decoder model, why the decoder model also used by the attention weights. On the first (c1), does that mean the decoder does not have context ? (i = ...
Jeremy Kenn's user avatar
0 votes
0 answers
21 views

understanding the distribution shift problem in direct preference optimization (DPO)

I'm having trouble understanding this paragraph of the DPO paper: Why does it matter so much that the preference data distribution aligns with the reference model output distribution? My ...
Ivy Cao's user avatar
0 votes
0 answers
13 views

Zero-shot out of distribution text classification

I'm building out a pipeline that would allow me to filter out text based on whether or not the text belongs to any of the classes I've defined. I feel like one (albeit naive) approach would simply be ...
mehsheenman's user avatar
0 votes
0 answers
36 views

Understanding the concepts of embedding in Roberta architecture?

I'm reading the implementation file of Roberta architecture, specifically in the RobertaEmbedding class, this class has the comment: ...
David's user avatar
  • 75
0 votes
1 answer
23 views

Why does skip-gram uses linear maps for embedding?

This question is inspired from Lilian Weng's blog here about skip-gram model, where she shows the model as multiplication via 2 matrices $W$ and $W'$ for embedding and word context matrix and $W$ ...
Mahammad Yusifov's user avatar
0 votes
1 answer
41 views

how can I interpret attention weights matrix? Are they reliable?

I've fine-tuned two different models (Bert and Roberta) on a dataset for a binary classification task and I'm comparing the sentences where the models predict wrong. I decided to use attention weights ...
Shayan's user avatar
  • 21
0 votes
0 answers
17 views

publically available language models that can be used to train arbitrary language data?

I have sentence data in a language that is not widely in use and as such popular LLMs do not support the language. I want to train some language model such that given some question, it is able to ...
Neijal Kanderbalt's user avatar
0 votes
0 answers
20 views

Are K and V values reused in each decoder layer's cross-Attention in the original "Attention is all you need" paper?

I'm working with Transformers and have a question about the encoder-decoder structure. In each decoder layer's cross-attention, are the K and V pairs from the corresponding encoder layer reused for ...
Dennis Yang's user avatar
2 votes
0 answers
32 views

How to learn text style in an article using LLMs?

What is the best way to learn text style in an article? By text style I mean special characteristics and patterns inherent to different authors/group's writing style. For-example, author attribution ...
Shayan's user avatar
  • 21
2 votes
1 answer
38 views

Why is the sinusoidal model classified as absolute positional encoding in some literature?

I am currently reading in depth about positional encodings, and as we know there are two types of positional encodings: Absolute and relative. My question: Why is the sinusoidal model classified as ...
Ali Haider Ahmad's user avatar
1 vote
0 answers
15 views

Neural Machine Translation with multi-language input to a single-language output?

I'm looking for NMT paradigms where the input to the model is the same text in N languages (e.g., L1, L2, L3) and the output is the translation in a different target language (e.g., L4). However, I ...
yigitcankaya's user avatar
0 votes
0 answers
375 views

Finetuning Mistral or MistralForSequenceClassification for text classification

I need to do text classification and have a dataset of 10K entries. I am considering using mistral and following a tutorial like https://huggingface.co/docs/transformers/training and replace model ...
Karl 17302's user avatar
0 votes
0 answers
11 views

What is the approximate minimum coding rate for NLP datasets?

I just realized that it is actually practical to use information theory to compute the maximum viable compression for datasets & that it is easiest to compute for discrete datasets. This makes me ...
profPlum's user avatar
  • 410
3 votes
3 answers
3k views

Do AI-based code-generators guarantee correct output?

I haven't spent much time looking at AI-based code-generators. What mechanism is used to generate code and how is it different than standard NLP?
FourierFlux's user avatar
0 votes
0 answers
44 views

Which input embeddings are learned during pre-training in BERT? What about during fine-tuning?

I was reading the 2019 BERT paper and they mention how they use wordpieces that are then represented as the sum of token embeddings, segment embeddings, and positional embeddings. What is unclear to ...
Karla's user avatar
  • 1
0 votes
0 answers
111 views

What is the loss function used when pre-training BERT on MLM & NSP tasks?

I'm new to NLP and was reading through the 2019 BERT paper and am confused about the loss function used during pre-training. As I understand it, the model is trained on the MLM and NSP tasks. The MLM ...
Karla's user avatar
  • 1
0 votes
0 answers
126 views

Why is there a "reference free" option in DPO (Direct Preference Optimization)'s loss function?

There is a reference_free parameter in trl's loss function implementation of DPO, while the original DPO paper does not mention the concept of "reference free". In trl's implementation: <...
Yang Bo's user avatar
  • 51
1 vote
1 answer
56 views

What would be the best AI model to query a database in natural language? [closed]

We want to build an AI Chatbot, that has access to our DB schema and the data (it is a RDBMS system) and can answer complex questions around this data. Few examples like - What was the most expensive ...
Atanu Roy's user avatar
  • 111
0 votes
1 answer
48 views

Using naive bayesian vs. transformer-based architecture model for human-annotated data?

I have a reddit dataset with thousands of online posts over the economy and inflation. We have used human-annotation on 60% of posts to determine whether users blame the following entities over the ...
maldini1990's user avatar
2 votes
0 answers
84 views

What is the meaning of "dimensionality of the embeddings and hidden states"?

I was reading the GPT-2 and LSTM documents and noticed that they use the terms "dimension of embedding and hidden state". For GPT-2, the size is $768$, and for LSTM, the size is $256$. What ...
user avatar
1 vote
1 answer
71 views

Are sentence-transformers the best available pretrained models for computing embeddings for a sentence?

I am playing around with the sentence transformers pretrained models to compute embeddings for a sentence and then compare it using the cosine similarity on the two vectors. I was using the ...
bigbang's user avatar
  • 113
1 vote
1 answer
75 views

Approaching construction of model that interprets financial reports [closed]

I want to train a model to be able to interpret financial reports (from a company). Basically, I want to be able to extract relevant information without needing to read through hundreds of pages of ...
Hlkwtz's user avatar
  • 11
2 votes
1 answer
144 views

Is beam search the actual obstacle that prevents GPT-style models from doing sophisticated math reasoning?

This is a rather soft question. Some people believe that GPT-style models can eventually solve very complex math problems if the models are large enough, but I'm skeptical about this. Suppose the GPT ...
Soha's user avatar
  • 21
1 vote
1 answer
571 views

Are there strictly deterministic LLMs?

LLMs are understood to generate non-deterministic outputs. My question is wether there are LLMs out there that are capable to producing deterministic outputs for any given input given fixed parameters ...
user599464's user avatar
0 votes
0 answers
127 views

Masking during Instruction Tuning for LLM finetuning

I'm currently trying to learn more about LLMs particularly generative decoder only models such as the GPT family of models. I do have one question about masking though. For me the way masking is ...
Pile_of_linear_Algebra's user avatar
0 votes
0 answers
23 views

Is Training Separate NER Models for Specific Labels a Viable Strategy for Improved Accuracy?

In my project, I'm utilizing Named Entity Recognition (NER) to identify crucial variables in prompts categorized into specific domains. However, incorporating numerous labels in the training data has ...
Raghul Azhagaiah's user avatar
0 votes
0 answers
44 views

Prefix tuning in LLM uses learnable vectors to fine tune the model

I would like to implement a new architecture for Transformer. Below description is my thought. Prefix tuning in LLM uses learnable vectors to fine tune the model. Is there a way to use the output ...
jackson's user avatar
1 vote
3 answers
69 views

Would maximizing (instead of minimizing) error of an LLM/HMM lead to complex behavior?

Imagine we have some sort of "next token predictor," either with transformer architecture, LSTM, or just a HMM (though the terminology I use here will be less aligned to HMMs, I believe the ...
BigMistake's user avatar
2 votes
1 answer
55 views

NLP "small" model to improve "big" model

When training the model for NLP is it important to get rid of data which has "bad semantic" for learning process? My plan is to create a "small model" which can decide whether data ...
Milkmaid's user avatar
  • 135
1 vote
0 answers
38 views

What is the input to an encoder-decoder transformer in next word prediction task?

I'm trying to understand how encoder-decoder architectures are used, or if they are used at all, for generative tasks that do not require an explicit prompt (ie. machine translation, summarization, ...
mehsheenman's user avatar
1 vote
1 answer
143 views

Masking in Decoder of Transformer

I understand that the masked multi-head attention block ensures that generation of token at time step t doesn't rely on subsequent tokens of the input. But the residual connection which adds the input ...
SAGALPREET SINGH's user avatar
0 votes
1 answer
42 views

If the unigram precision is (N-1)/N, then the bigram precision is :

Consider the following machine translation scenario. The reference translation has N words (do not consider sentence beginner ‘hat’ and sentence finisher ‘dot’). The machine output also has N words. ...
Geeklovenerds's user avatar
0 votes
1 answer
122 views

what are the applications scenarios for prefix decoder LMs

Motivated by this post wherein one of the comments mentioned the use-case for encoder-decoder LM. I wanted to know when to use prefix-decoder LM? vis a vis encoder-decoder or causal decoder only ...
Singh's user avatar
  • 1
1 vote
0 answers
26 views

How to measure similarities between text (word to word or word to phrase)?

Is there a way to measure to measure similarities between two pieces of text ? Think about the case where you have an image captioning model but you only want to deal/use specific class names. e.g. ...
Souhaielrmx's user avatar
0 votes
2 answers
103 views

Why encoders are required in Transformers

In the original Transformers paper why encoder is added when a decoder alone can do what an encoder can do (like multi-head attention, feed-forward NN etc....). I mean even a decoder also has the same ...
Swastik's user avatar
0 votes
1 answer
54 views

Setting number of rows returned by vector stores

When using vector stores like pinecone or Faiss from langchain, is it possible to set the number of records returned based on similarity search? For example, consider the following code, is there a ...
Karl 17302's user avatar
0 votes
0 answers
20 views

Exploring the Similarity of Sibling’s Voices Using Automatic Speaker Recognition

I want to start project on Exploring the Similarity of Sibling’s Voices Using Automatic Speaker Recognition Everyone has a unique voice, because of the different structure of their articulatory ...
Alan Turing's user avatar
1 vote
0 answers
130 views

LLM for Postgres

I have a postgres database with 200+ Tables. Each table contains information about my supply inventory. It also contains columns which are JSON and there are nested JSON as well. There are ...
Shivkumar Mallesappa's user avatar
2 votes
1 answer
365 views

Aren't context lengths for transformers an artificial restriction?

Let's focus on the case of decoder-only transformers, where I am using algorithm 10 from "Formal Algorithms for Transformers" by Mary Phung and Marcus Hutter as a reference. : https://i....
Robert Wegner's user avatar
1 vote
0 answers
25 views

Is there any well-established work that allows robots to communicate their decision-making using natural language?

I am searching for a well-established work that allows robots to communicate their decision-making using natural language. For example, a robot's explanation could be "I did [task1] because [...
jigz's user avatar
  • 21
0 votes
0 answers
160 views

Is ChatGPT a viable strategy for solving the P-NP Problem?

According to ZDNet, it is an open question whether a transformer LLM like ChatGPT can facilitate the determination of a solution to the P-NP Problem. (See Can generative AI solve computer science's ...
J D's user avatar
  • 101
0 votes
1 answer
44 views

How to train a seq2seq model to rephrase input text following given rules

I want to train (fine-tune) a seq2seq model to perform the task of rephrasing input following these rules : 1- always follow the pattern "Entity Verb Entity" 2- only use simple sentences : ...
Wissem Boujlida's user avatar

1
2 3 4 5
16