Questions tagged [natural-language-processing]

For questions related to natural language processing (NLP), which is concerned with the interactions between computers and human (or natural) languages, in particular how to create programs that process and analyze large amounts of natural language data.

Filter by
Sorted by
Tagged with
8 votes
1 answer
813 views

Are there transformer-based architectures that can produce fixed-length vector encodings given arbitrary-length text documents?

BERT encodes a piece of text such that each token (usually words) in the input text map to a vector in the encoding of the text. However, this makes the length of the encoding vary as a function of ...
1 vote
1 answer
1k views

What is considered the pre-fill, and what is considered the decoding phase in this process?

I've seen conflicting information about this online so I'm looking for clarification. I'm dealing with the causal LLaMAF model specifically. I used to think that a sequence of tokens is generated in, ...
1 vote
2 answers
58 views

What technique is used for training Large Language Models like GPT?

I'm learning about GenAI, such as GPT (Generative Pretrained Transformer), and I'm particularly interested in understanding the training techniques used for these models. Deep learning, generally, can ...
1 vote
1 answer
54 views

Why do Transformer decoders use masked self attention when producing new tokens?

I've been reading that transformer decoders use masked self attention so that the decoder can't cheat by looking ahead. For example, when predicting the 6th token in the sequence we shouldn't have ...
-1 votes
0 answers
11 views

What is the best language model for summarising legal documents? [closed]

I have a set of >20,000 legal documents that I want to summarise key details of. For example, parties, transaction object, transaction price, geographic scope, etc. Currently I'm using the OpenAI ...
0 votes
2 answers
164 views

If I freeze pre-trained model weights and than train a classifier on top of its embeddings does that called fine-tunning?

In the context of machine learning. If I freeze pre-trained model weights (for example, BERT) and then train a classifier on top of its embeddings, does that called fine-tuning?
0 votes
1 answer
101 views

How to configure a neural network to selectively change only certain characters in a string?

I'm trying to figure out how to train a neural network to macronize Latin text. Essentially, in Latin, vowels can either be long or short, and length is indicated with a macronized character: i.e. o ...
0 votes
0 answers
25 views

How using sinusoidal encoding for positional encoding saves space in comparison to binary encoding? [closed]

Context: The blog - https://kazemnejad.com/blog/transformer_architecture_positional_encoding/ mentions that using binary encoding would waste the space. Using the ...
0 votes
0 answers
11 views

Using a HMM for grammar checking

I want to try to make an HMM from scratch for POS tagging, which I would extend to grammar checking. I understand there's much better ways for a grammar checker but this is just as a learning ...
0 votes
1 answer
33 views

What language model can convert normal text to JSON data

I have tried training T5-small, T5-base and T5-Large on around 15K rows of data where input data was something like but I did not get desired results ...
2 votes
1 answer
38 views

Why is the sinusoidal model classified as absolute positional encoding in some literature?

I am currently reading in depth about positional encodings, and as we know there are two types of positional encodings: Absolute and relative. My question: Why is the sinusoidal model classified as ...
0 votes
1 answer
475 views

How big the context can be using HuggingFace models?

I'm new on AI, Neural Networks, ChatBots and all this ecosystem. I'm trying to use a classical example of pre-trained models, more specifically ...
1 vote
1 answer
82 views

Why do current language models no longer generate to long or short texts?

One of the biggest strengths of ChatGPT is that it generates fitting text with respect to the input query. It usually stays on topic, anwers the question completely and especially does not start ...
0 votes
1 answer
41 views

how can I interpret attention weights matrix? Are they reliable?

I've fine-tuned two different models (Bert and Roberta) on a dataset for a binary classification task and I'm comparing the sentences where the models predict wrong. I decided to use attention weights ...
0 votes
0 answers
22 views

Where I can find SOTA models of information retrieval?

Can someone tell me where I can find SOTA models of information retrieval? My task is to rank documents by given query by semantic search of embedding. I know that models like ColBERT, SPLADE solve ...
0 votes
1 answer
81 views

Semantic-based evaluation of translations instead of BLEU

I have a text generation model and I want to evaluate its output by comparing it to a set of gold human-annotated references. I went through machine-translation metrics and I found that BLEU is used ...
0 votes
2 answers
5k views

How to fine-tune GPT-J with small dataset

I have followed this guide as closely as possible: https://github.com/kingoflolz/mesh-transformer-jax I'm trying to fine-tune GPT-J with a small dataset of ~500 lines: ...
4 votes
1 answer
2k views

Why aren't the BERT layers frozen during fine-tuning tasks?

During transfer learning in computer vision, I've seen that the layers of the base model are frozen if the images aren't too different from the model on which the base model is trained on. However, on ...
24 votes
5 answers
11k views

Why does ChatGPT fail in playing "20 questions"?

IBM Watson's success in playing "Jeopardy!" was a landmark in the history of artificial intelligence. In the seemingly simpler game of "Twenty questions" where player B has to ...
0 votes
0 answers
24 views

Why won't ChatOllama work with GraphCypherQAChain?

I am trying to get a QA Chain that uses mistral to work, but the Chain fails to restrict the LLM to solely use the relationships in the Knowledge Graph (in Neo4j). The code is: ...
0 votes
0 answers
16 views

Does the fixed context in attention mechanism is accquired after getting the decoder hidden layer of the first hidden state?

here, the fixed context vector (ci) is used for the decoder model, why the decoder model also used by the attention weights. On the first (c1), does that mean the decoder does not have context ? (i = ...
0 votes
0 answers
21 views

understanding the distribution shift problem in direct preference optimization (DPO)

I'm having trouble understanding this paragraph of the DPO paper: Why does it matter so much that the preference data distribution aligns with the reference model output distribution? My ...
5 votes
3 answers
5k views

Can I always use "encoding" and "embedding" interchangeably?

This question is restricted to the text domain only. The meaning of the word "encode" is Convert (information or instruction) into a particular form. One which performs encoding is called an ...
1 vote
1 answer
56 views

What would be the best AI model to query a database in natural language? [closed]

We want to build an AI Chatbot, that has access to our DB schema and the data (it is a RDBMS system) and can answer complex questions around this data. Few examples like - What was the most expensive ...
4 votes
1 answer
4k views

Why are GAN models not heavily used for NLP?

I am wondering why there has not been more usage of GANs for NLP. I know there has been research on the subject (The Google Scholar page for the subject is here). Are there any specific reasons why ...
0 votes
0 answers
13 views

Zero-shot out of distribution text classification

I'm building out a pipeline that would allow me to filter out text based on whether or not the text belongs to any of the classes I've defined. I feel like one (albeit naive) approach would simply be ...
0 votes
0 answers
36 views

Understanding the concepts of embedding in Roberta architecture?

I'm reading the implementation file of Roberta architecture, specifically in the RobertaEmbedding class, this class has the comment: ...
1 vote
2 answers
182 views

How do I create syntactically correct sentences given several words?

Is there an AI application that can produce syntactically (and semantically) correct sentences given a bag of words? For example, suppose I am given the words "cat", "fish", and "lake", then one ...
0 votes
1 answer
23 views

Why does skip-gram uses linear maps for embedding?

This question is inspired from Lilian Weng's blog here about skip-gram model, where she shows the model as multiplication via 2 matrices $W$ and $W'$ for embedding and word context matrix and $W$ ...
0 votes
1 answer
271 views

Hot to calculate Maximum Normalized log Probability for Active Learning with BERT

I have encountered difficulties understanding the calculation of Maximum Normalized Log Probabilities acording to Shen et al.. With n being the sequence length, yi the label of word i. Xij is the ...
0 votes
1 answer
117 views

Can you make Q&A language model stay on topic?

I’m thinking of fine-tuning a pre-trained language model for a Q&A task. More specifically, I’d like to fine-tune the model on a single chapter in a classic college textbook. Afterward, the reader ...
6 votes
1 answer
2k views

Why does ChatGPT create fake code?

ChatGPT has been a big thing lately. It also makes a lot of mistakes. For example, it creates fake functions of a package and tells it as it works for real. I was wondering how that works. Why is it ...
0 votes
0 answers
17 views

publically available language models that can be used to train arbitrary language data?

I have sentence data in a language that is not widely in use and as such popular LLMs do not support the language. I want to train some language model such that given some question, it is able to ...
0 votes
0 answers
20 views

Are K and V values reused in each decoder layer's cross-Attention in the original "Attention is all you need" paper?

I'm working with Transformers and have a question about the encoder-decoder structure. In each decoder layer's cross-attention, are the K and V pairs from the corresponding encoder layer reused for ...
0 votes
1 answer
48 views

Using naive bayesian vs. transformer-based architecture model for human-annotated data?

I have a reddit dataset with thousands of online posts over the economy and inflation. We have used human-annotation on 60% of posts to determine whether users blame the following entities over the ...
2 votes
0 answers
32 views

How to learn text style in an article using LLMs?

What is the best way to learn text style in an article? By text style I mean special characteristics and patterns inherent to different authors/group's writing style. For-example, author attribution ...
1 vote
1 answer
61 views

Is image machine translation done in two steps?

Suppose I have images of hand-written Japanese text. If I want to translate those images, would my ML algorithm be a 2-step model (for example, a CNN to convert the image into Japanese characters/...
1 vote
0 answers
15 views

Neural Machine Translation with multi-language input to a single-language output?

I'm looking for NMT paradigms where the input to the model is the same text in N languages (e.g., L1, L2, L3) and the output is the translation in a different target language (e.g., L4). However, I ...
0 votes
2 answers
119 views

Which NLP applications are based on recurrent neural networks?

Some of the NLP applications taken from this link NLP Applications: Machine Translation Speech Recognition Sentiment Analysis Question Answering Automatic Summarization Chatbots Market Intelligence ...
3 votes
2 answers
2k views

Why does CLIP use a decoder-only transformer for encoding text?

In CLIP [1], the authors train a model to learn multi-modal (text, vision) embeddings by maximizing the cosine similarity between text and image embeddings produced by text and image encoders. For the ...
0 votes
3 answers
793 views

How much labelling is required for NER with SpaCy?

I have transaction data and I would like to extract the merchant from the transaction description. I am new to this but I just came across Named Entity Recognition and SpaCy. I have hundreds of ...
1 vote
2 answers
85 views

Convolutional Sequence to Sequence Learning kernel parameters

I am reading the paper Convolutional Sequence to Sequence Learning by Facebook AI researchers and having trouble to understand how the dimensions of convolutional filters work here. Please take a look ...
0 votes
1 answer
59 views

What are examples of simple gradient based NLP models?

I am looking for a simple example of gradient-based methods for NLP. More specifically I am looking for post-hoc local explanations gradient-based methods, that is to say, which explain a single ...
0 votes
1 answer
1k views

In-batch negative training Improves the results

I have read Dense passage retrieval for Open Domain Question Answering, and in page 6 it talks about in-batch negative training, it states the following: We find that using a similar configuration (7 ...
0 votes
0 answers
375 views

Finetuning Mistral or MistralForSequenceClassification for text classification

I need to do text classification and have a dataset of 10K entries. I am considering using mistral and following a tutorial like https://huggingface.co/docs/transformers/training and replace model ...
3 votes
3 answers
3k views

Do AI-based code-generators guarantee correct output?

I haven't spent much time looking at AI-based code-generators. What mechanism is used to generate code and how is it different than standard NLP?
0 votes
0 answers
11 views

What is the approximate minimum coding rate for NLP datasets?

I just realized that it is actually practical to use information theory to compute the maximum viable compression for datasets & that it is easiest to compute for discrete datasets. This makes me ...
6 votes
4 answers
1k views

Can Machine Learning be applied to decipher the script of lost ancient languages?

Can Machine Learning be applied to decipher the script of lost ancient languages (namely, languages that were being used many years ago, but currently are not used in human societies and have been ...
0 votes
0 answers
44 views

Which input embeddings are learned during pre-training in BERT? What about during fine-tuning?

I was reading the 2019 BERT paper and they mention how they use wordpieces that are then represented as the sum of token embeddings, segment embeddings, and positional embeddings. What is unclear to ...
0 votes
1 answer
59 views

NLP: What is expected from the output of a perfect coreference system?

For instance, consider the following piece of text: 'The father of Richard is a very nice guy. He was born in a poor family. Because of that, Richard learnt very good values. Richard is also a very ...

1
2 3 4 5
16