Skip to main content

Questions tagged [large-language-models]

The tag has no usage guidance, but it has a tag wiki.

Filter by
Sorted by
Tagged with
0 votes
1 answer
64 views

how can I interpret attention weights matrix? Are they reliable?

I've fine-tuned two different models (Bert and Roberta) on a dataset for a binary classification task and I'm comparing the sentences where the models predict wrong. I decided to use attention weights ...
Shayan's user avatar
  • 21
1 vote
1 answer
32 views

publically available language models that can be used to train arbitrary language data?

I have sentence data in a language that is not widely in use and as such popular LLMs do not support the language. I want to train some language model such that given some question, it is able to ...
Neijal Kanderbalt's user avatar
0 votes
0 answers
15 views

Are There Scientific Papers on Methods for Robustifying LLMs and Current Challenges They Face?

I'm currently researching Large Language Models (LLMs) and am particularly interested in the recent advancements and challenges in this field. My focus is on understanding the methods being developed ...
Iman Mohammadi's user avatar
1 vote
2 answers
310 views

Which techniques are best suitable for explainable AI for LLM models

I am currently working with large language models like llama and mistral, interested in techniques for making these models more explainable. I am looking for some tools or techniques which can help me ...
Hiren Namera's user avatar
2 votes
0 answers
37 views

How to learn text style in an article using LLMs?

What is the best way to learn text style in an article? By text style I mean special characteristics and patterns inherent to different authors/group's writing style. For-example, author attribution ...
Shayan's user avatar
  • 21
1 vote
0 answers
17 views

Neural Machine Translation with multi-language input to a single-language output?

I'm looking for NMT paradigms where the input to the model is the same text in N languages (e.g., L1, L2, L3) and the output is the translation in a different target language (e.g., L4). However, I ...
yigitcankaya's user avatar
10 votes
3 answers
4k views

Why are LLMs able to reproduce bodies of known text exactly?

Mathematically, I wouldn't expect LLMs to be able to reproduce source texts exactly unless the source text was the probable outcome given some prompt. However, I have now tested HuggingFaceH4/zephyr-...
Grant Curell's user avatar
1 vote
1 answer
43 views

Comparing the performances of GPTs with deep learning in the field of binary files and their related reports

Regarding the case study of a dataset including binary files (containing assembly code) and reports related to each file (the content of the static analysis of the file as well as the analysis of the ...
user16385455's user avatar
1 vote
0 answers
669 views

Finetuning Mistral or MistralForSequenceClassification for text classification

I need to do text classification and have a dataset of 10K entries. I am considering using mistral and following a tutorial like https://huggingface.co/docs/transformers/training and replace model ...
Karl 17302's user avatar
1 vote
0 answers
35 views

Why doesn't CLIP use a pretrained large language model as the text encoder?

CLIP: https://openai.com/research/clip They use a small text encoder and suggest that the simpler the model the better. Is there any reason why that is? Has anyone tried using a pretrained LLM as the ...
JobHunter69's user avatar
3 votes
1 answer
2k views

Do LLMs based on a diffusion model (as opposed to an autoregressive model) exist?

Is there a such thing (described in title), at least in research papers (not actual models)? So far all LLMs that I know are autoregressive models.
user avatar
3 votes
3 answers
3k views

Do AI-based code-generators guarantee correct output?

I haven't spent much time looking at AI-based code-generators. What mechanism is used to generate code and how is it different than standard NLP?
FourierFlux's user avatar
0 votes
0 answers
16 views

which the effective ranks of embedding matrices in Large LMs?

The hidden size of current LLMs is well in the 8K and above, and I was wondering, has the rank of the embedding matrix, and the rank of the transpose of the output matrix, been increasing with the ...
arivero's user avatar
  • 51
2 votes
0 answers
35 views

Why are text models instructed in second person perspective?

Many text model system prompts begin with "You are an AI assistant. You are helpful. You …. [user input] [model output]", but looking at what might have been in the training material, I ...
allo's user avatar
  • 310
0 votes
0 answers
64 views

Can the methods of FunSearch that have been applied to the cap set problem be also applied to the enumeration of maximal antichains of sets?

Today on a seminar we discussed with colleagues recent impressive breakthrough in the cap set problem using Google DeepMind's FunSearch. The following specific question has been raised. Among several ...
მამუკა ჯიბლაძე's user avatar
0 votes
1 answer
122 views

How can a LLM optimise on it's own processing speed

Is it possible to give a large-language-model access to the time it takes to give it's answers and ask it to optimize on that? An example prompt would be: From now on, please measure the time it ...
Fabian Zeindl's user avatar
0 votes
0 answers
219 views

Why is there a "reference free" option in DPO (Direct Preference Optimization)'s loss function?

There is a reference_free parameter in trl's loss function implementation of DPO, while the original DPO paper does not mention the concept of "reference free". In trl's implementation: <...
Yang Bo's user avatar
  • 51
1 vote
1 answer
98 views

Will open LLM be subject to AI act / considered High Risk?

As I was reading the AI act I was wondering how it will apply to current open LLM (say Mixtral of Mistal AI). It seems that there are some specific arguments regarding open LLM that would put them ...
Lucas Morin's user avatar
0 votes
1 answer
616 views

How does Chat GPT encode a question?

Chat GPT is based on a decoder-only Transformer so it does not have an encoder. Given that, how is a user's question passed as input to Chat GPT's decoder? In a regular encoder-decoder architecture, ...
joan's user avatar
  • 1
1 vote
1 answer
76 views

Approaching construction of model that interprets financial reports [closed]

I want to train a model to be able to interpret financial reports (from a company). Basically, I want to be able to extract relevant information without needing to read through hundreds of pages of ...
Hlkwtz's user avatar
  • 11
1 vote
0 answers
52 views

Largest fully open source LLMs including training

It seems that most of the open-sourced large language models (LLMs) like Llama 2 had the model released but not the exact training procedure and training data-sources (exact data revisions) so that ...
Kozuch's user avatar
  • 281
1 vote
1 answer
66 views

What is the current limit for LLMs regarding the size of the context that can be fed to them?

Is there a limitation in current large language models (LLMs) in terms of practical processing time or memory resources when it comes to digesting the context provided by users? What I mean regarding ...
Miki's user avatar
  • 111
3 votes
1 answer
207 views

Is beam search the actual obstacle that prevents GPT-style models from doing sophisticated math reasoning?

This is a rather soft question. Some people believe that GPT-style models can eventually solve very complex math problems if the models are large enough, but I'm skeptical about this. Suppose the GPT ...
Soha's user avatar
  • 31
5 votes
2 answers
1k views

Are there strictly deterministic LLMs?

LLMs are understood to generate non-deterministic outputs. Are there LLMs out there that are capable to producing deterministic outputs for any given input given fixed parameters (like e.g temperature)...
user599464's user avatar
0 votes
0 answers
112 views

Intentionally corrupting LLM weights (lobotomy)

It is largely unknown how LLMs work inside. Has anyone scientifically tried to corrupt (open source) model's weights in an organized manner to maybe detect which parts of the model are doing what or ...
Kozuch's user avatar
  • 281
0 votes
0 answers
367 views

Masking during Instruction Tuning for LLM finetuning

I'm currently trying to learn more about LLMs particularly generative decoder only models such as the GPT family of models. I do have one question about masking though. For me the way masking is ...
Pile_of_linear_Algebra's user avatar
0 votes
0 answers
20 views

Could an analysis of GPT4's WAIS score be published?

Similar to this: https://www.scientificamerican.com/article/i-gave-chatgpt-an-iq-test-heres-what-i-discovered/ But more detailed and in depth (subtest breakdown, including image analysis, etc.), WAIS-...
BigMistake's user avatar
1 vote
1 answer
1k views

How LLM keeps the context of a chat/thread

How an LLM keeps the context (what has already been entered by the user) of a chat/thread? For reference, in chat.openai.com, for each chat we create (or a Thread according to their API), the LLM ...
Wenuka's user avatar
  • 111
0 votes
0 answers
121 views

Specifying multiple sources of RAG in different formats

I am considering whether to do RAG or Fine tuning for LLM to answer questions for clients in a small business The small business is a car rental company that gets new clients call up about certain ...
dowjones123's user avatar
0 votes
0 answers
32 views

Requirements for AI to be able to read a book and accurately recall the facts?

As a fun random project I've been working on is learning languages and making cheatsheets for them, dictionaries, etc.. I would love to encode the grammar rules which have been captured in books ...
Lance's user avatar
  • 153
2 votes
1 answer
475 views

Why are some LLMs trained on both CommonCrawl and Wikipedia/StackExchange?

Some LLMs are trained on both CommonCrawl and Wikipedia/StackExchange. Why? Does CommonCrawl already contain Wikipedia/StackExchange? E.g., from the LLaMa 1 paper: and from https://huggingface.co/...
Franck Dernoncourt's user avatar
0 votes
0 answers
58 views

From Text-To-Speech to LLMs: Providing "writing style"

I've just recently learned about Text-To-Speech models and how they are trained. Unlike LLMs, to a provided pair (text, speech), a feature vector ${f}$, that was generated by more speech of that ...
Mathy's user avatar
  • 153
2 votes
1 answer
145 views

LLM Hallucinations on In-Context Data

I know of some benchmarks that LLMs do undergo, but I am no expert whatsoever. I think what I am wondering about is closest to TruthfulQA. The question came up when I heard of combining company data ...
Mathy's user avatar
  • 153
1 vote
1 answer
83 views

How can LLMs understand and perform meta tasks? (e.g. summarization)

I don't ask how to make it summarize xy but if it is known how a "LLM" understands and performs this meta task at all. The same is true for prompts like "Be brief" or "Explain ...
d3nigma's user avatar
  • 23
0 votes
0 answers
36 views

How does VRAM usage change when running prompts async through LLMs?

If my LLM uses X GB of VRAM when processing prompts sequentially, does it use 2X GB if I run 2 prompts at once using async support? e.g. using Langchains ...
codeananda's user avatar
1 vote
0 answers
168 views

LLM for Postgres

I have a postgres database with 200+ Tables. Each table contains information about my supply inventory. It also contains columns which are JSON and there are nested JSON as well. There are ...
Shivkumar Mallesappa's user avatar
0 votes
0 answers
70 views

claude 2 doesn't know basic math? [duplicate]

Sometimes, when I see answers like this from large language models, it makes me feel disgusted: Me: Does Voyager 1 have enough velocity to escape the solar system without using Jupiter's gravity ...
Mr Saw's user avatar
  • 1
3 votes
0 answers
130 views

How do I balance context and history when creating prompts for LLM's?

A conversation through the OpenAI API looks something like this ...
Ian Purton's user avatar
0 votes
0 answers
218 views

Is ChatGPT a viable strategy for solving the P-NP Problem?

According to ZDNet, it is an open question whether a transformer LLM like ChatGPT can facilitate the determination of a solution to the P-NP Problem. (See Can generative AI solve computer science's ...
J D's user avatar
  • 101
0 votes
1 answer
63 views

How to train a seq2seq model to rephrase input text following given rules

I want to train (fine-tune) a seq2seq model to perform the task of rephrasing input following these rules : 1- always follow the pattern "Entity Verb Entity" 2- only use simple sentences : ...
Wissem Boujlida's user avatar
-1 votes
1 answer
660 views

How to get Llama-2 Rotary Embeddings?

I want to get the Llama-2 rotary embeddings. I do print(model) and get the following output: In the picture I highlight the rotary embeddings. How can get the ...
Christian01's user avatar
0 votes
0 answers
30 views

What options do I have when performing RAG on similarly phrased chunks?

I'm building a RAG pipeline to extract real estate phrases from excel documents. These phrases are short (2-5 words) and are often phrased differently. I've manually added in different phrases to each ...
codeananda's user avatar
0 votes
0 answers
22 views

What is the policy model in RLHF for LLMs?

What is the policy model doing explicitly in an LLM with RLHF setup? From my understanding, LLMs generate in a way that is no different from any of their predecessors: beam search decoding, ...
information_interchange's user avatar
2 votes
2 answers
3k views

How can I teach a book to an LLM?

I am trying to find out how I can teach the content of a whole, multiple hundert pages book to an LLM so that it "knows" all details and can be queried, give summaries etc. The book is one ...
dschuld's user avatar
  • 21
3 votes
2 answers
132 views

Why do current language models no longer generate to long or short texts?

One of the biggest strengths of ChatGPT is that it generates fitting text with respect to the input query. It usually stays on topic, anwers the question completely and especially does not start ...
Ricu's user avatar
  • 31
1 vote
2 answers
510 views

Tips and tricks when training a very large language model?

Have never trained a (very) large language model, so I am wondering if the process is the same as training a (regular) language model, i.e. you prepare the data, set up the architecture, ...
information_interchange's user avatar
0 votes
1 answer
176 views

How to prompt an LLM for a real variable list length? [closed]

I struggle to get a variable list out of an LLM. Even with a prompt like ...
flori's user avatar
  • 109
-2 votes
2 answers
97 views

How successful are the state-of-the-art (2023) email filters really? [closed]

How successful are the state-of-the-art (2023) email filters really? Some references claim that spam detection may reach high accuracy in test settings, but I've thought that email filtering should ...
mavavilj's user avatar
  • 105
2 votes
1 answer
924 views

Why do transformers compute the loss over the prompt?

When fine-tuning large language models, which are commonly decoder transformer architectures, sometimes we want to compute the training loss over the entire prompt, sometimes just the completion ...
milez's user avatar
  • 981
0 votes
0 answers
151 views

Does Embeddings and Vector Databases solve the need of having longer context windows?

I am learning to use the OpenAI API to build LLM-based agents. I recently came across the concept of vector databases, which use embeddings to convert text into vectors and store them in a database ...
Cesar Ruiz's user avatar