Skip to main content
Share Your Experience: Take the 2024 Developer Survey

Questions tagged [large-language-models]

The tag has no usage guidance, but it has a tag wiki.

Filter by
Sorted by
Tagged with
0 votes
0 answers
23 views

Are there LLMs that are allowed time to contemplate freely?

We can call it "time to think", or "free contemplation", or whatever. Maybe a term already exists in the field? What I mean is - with so many LLMs being "trained" on ...
Mentalist's user avatar
  • 101
0 votes
0 answers
24 views

Why fine tuning does not work as well as RAG?

I cannot find a definite answer to this question. Suppose I want to build a QA (question answering) system on a set of personal documents. It looks that RAG (retrieval augmented generation) is the way ...
Thomas's user avatar
  • 163
1 vote
1 answer
36 views

What should be Relationship between embedding dimension and context length?

What should we keep hidden dimension/embedding dimension (d_model as per attention is all you need paper), greater, equal, or smaller to the context length (n)? Is there any such relationship between ...
Aamod Thakur's user avatar
0 votes
0 answers
11 views

Does a vector database maintain pre-vector chunked data for RAG systems?

I believe that when using an LLM with a Retrieval-Augmented Generation (RAG) approach, the results retrieved from a vector search must ultimately be presented in text form. Otherwise, the prompt would ...
shinyatk's user avatar
  • 101
0 votes
0 answers
10 views

Best Searchbots that explain code

Of all the LLM based available searchbots which ones are best at explaining code from Java, Python, Golang or Javascript? ChatGPT, Bing Copilot, Claude, Groq or Le Mistral?
user721025's user avatar
0 votes
0 answers
17 views

How is a LLM able to override its prior knowledge through In-Context Learning?

I came across a Google's blog (https://research.google/blog/larger-language-models-do-in-context-learning-differently/) discussing large language models (LLMs) and how we can overried their prior ...
tidealwari91's user avatar
0 votes
0 answers
14 views

Any LMs that use tanh (generalization) instead of sigmoid within Attention?

Question is in the title. Posts such as this and this mention how this would be possible. I have some colleagues who have anecdotally heard of this being done on very small transformer models but I ...
naston's user avatar
  • 11
0 votes
0 answers
14 views

What's the action space in RLHF for LLM?

I've been trying to understand how the modern LLMs use PPO for fine-tuning. In the PPO algorithm, one has to compute advantages, which are then used for either increasing or decreasing action's ...
Druudik's user avatar
  • 149
2 votes
1 answer
57 views

Can attention models be replaced by non-sigmoid activation functions?

As far as I understand, the attention model in a LLM is used to mitigate the vanishing gradient problem. When using activation functions like the sigmoid function, deep neural networks may lead to ...
A. Darwin's user avatar
  • 133
0 votes
1 answer
22 views

Reference request: data efficiency of LLM pre-training

I've seen it stated multiple times that LLMs have much worse data efficiency than humans (IE require more data to reach same or worse performance), EG this Tweet by Yann LeCun, or 19:30 in this talk ...
Jake Levi's user avatar
  • 101
1 vote
1 answer
68 views

Is there a relationship between tokens and parameters in LLMs?

What the question says. In a transformer architecture, is there a relationship between number of tokens and number of parameters? Can you have a LLM with a small number of parameters but a large ...
A. Darwin's user avatar
  • 133
0 votes
0 answers
34 views

Is there research on the effect of typos in LLM prompts?

A simple typo can split a single token for a common word into several tokens, not only making the prompt longer, but also creating a combination of tokens that was rare in the training set. I wonder ...
allo's user avatar
  • 310
0 votes
1 answer
28 views

How to create 1 embedding for text + image

I'm using Ollama to run llm's. I can create embeddings for text and images, which I store in ChromaDB. The goal of all this, is to find content which best fits a question, so I can create a good ...
Jeanluca Scaljeri's user avatar
1 vote
0 answers
15 views

RAG can be done using langchain, Now CohereForAI's Command R +, provides similar functionality, how it is is better than langchain's implementation? [closed]

How langchain's RAG implementation is different from the CohereforAI(Command R+) RAG functionality and what edge does it provide over the langchain RAG implementation. I tried both of them, But I am ...
Andhoju Karthikeya's user avatar
0 votes
0 answers
8 views

How to Measure the Impact of Adding a Translation Layer to Large Language Models?

I'm currently exploring the integration of a translation layer into Large Language Models (LLMs) to preprocess all inputs into a uniform language (English) before processing by the model. The idea is ...
Ilya Gazman's user avatar
0 votes
0 answers
13 views

From how many experts does LLM training using a mixture of experts (MoE) start slowing down compared to training LLMs without MoE?

I'm trying to find some information regarding the impact of the number of experts on LLM training. From how many experts does LLM training using a mixture of experts (MoE) start slowing down compared ...
Franck Dernoncourt's user avatar
0 votes
1 answer
73 views

Train my own LLM on a smaller corpus of text?

Would it be possible to train my own LLM on a smaller corpus of text, lets say some coding documentation that I then want to ask questions about using the model? If so, are there any recommended ways ...
Dylan Dijk's user avatar
2 votes
1 answer
40 views

Could LLMs perform the autoregressive generation with probability vectors instead of choosing a discrete token every time?

As I understand it, GPT-style LLMs take a sequence of tokens as input and output a token probability vector. The first thing that happens to an input token is that it goes through the input embedding, ...
MelonDude's user avatar
0 votes
1 answer
78 views

LLMs as "fuzzy JPEGs"

We should conceptualize LLMs as very quirky and very experimental if advanced information retrieval systems (fuzzy JPEGs) according to point 7 in the screenshot below (grabbed from pmarca's Twitter ...
thanks_in_advance's user avatar
0 votes
0 answers
16 views

Problems with understanding instruction fine-tuning

I'm trying to read up on instruction fine-tuning, but I think I have a big misunderstanding. As I understand, instruction datasets typically have 3 components: (a) the instruction (b) the output/...
Christian's user avatar
  • 101
1 vote
0 answers
100 views

Is Claude-3-Haiku a mixture of experts model?

If it is not, it is a significant step forward as it is at the level of the original GPT-4, which is a mixture of experts.
Anixx's user avatar
  • 331
0 votes
1 answer
57 views

Can GenAI be Used to Generate Decision Trees from Text?

I'm exploring the capabilities of GenAI for text analysis and decision-making processes. I'm particularly interested in understanding whether GenAI can be leveraged to create decision trees directly ...
D3N3ON's user avatar
  • 1
0 votes
0 answers
39 views

How to teach Gemma model my mother tongue (Kannada - one of the oldest Indic languages)

I'm interested in teaching the Gemma 2B model my mother tongue (Kannada - one of the oldest Indic languages). The pre-trained model doesn't work well with the mentioned language, so I thought of ...
Swastik's user avatar
  • 101
0 votes
0 answers
14 views

What is the current literature on "appending" an entire LLM model as part of the knowledge another LLM can use?

I've been studying retrieval augmented generation and vector databases recently. In a nutshell, vector based retrieval works by first dividing up input data(whether it's a string, image, etc) into ...
rasputin's user avatar
0 votes
0 answers
21 views

What's the current best ways to extend context windows in LLMs?

What's the state of the art? The most elegant/ simplest to implement? The ones that require least computational resources?
JobHunter69's user avatar
0 votes
0 answers
76 views

Why do LLM tokenizers use a special symbol for space such as Ġ in BPE or ▁ in SPM?

Popular tokenizers use a special symbol such as "Ġ" (BPE) or "▁" (SentencePiece) to represent space. What is the reasoning behind this? I did try searching for the answer. I got ...
Borislav Stanimirov's user avatar
1 vote
0 answers
42 views

Why are Mistral LLMs branded as enabling RAG-QA (Retrieval-Augmented Generation for Question Answering)?

I read on https://mistral.ai/news/mistral-large/: Mistral Small benefits from the same innovation as Mistral Large regarding RAG-enablement and function calling. What do they mean by RAG-enablement? ...
Franck Dernoncourt's user avatar
0 votes
1 answer
57 views

Minimum Number of Samples for LLM Benchmark?

I am working on a project to evaluate various fine-tuned LLMs. Unfortunately inference is prohibitively slow, and I don't think I will be able to test my models on the full test set of 40,000 samples. ...
Jonathan Emmons's user avatar
0 votes
1 answer
66 views

How do I code so that the embedding output and input share the same weight matrices?

I am trying to implement the Attention is All You Need paper from scratch. The authors mentioned in section 3.4 that "In our model, we share the same weight matrix between the two embedding ...
OneMoreGamble's user avatar
1 vote
2 answers
112 views

How are sentences turned into a vector in LLM

My understanding of Large Language Models like GPT is that they are special kinds of deep neural networks specifically trained to predict the next word, given the beginning of a sentence. I understand ...
Weier's user avatar
  • 131
0 votes
2 answers
112 views

What is an "inference kernel"?

In the recent event where ChatGPT "went crazy", this term was used in the official post-mortem to describe what happened: In this case, the bug was in the step where the model chooses these ...
hippietrail's user avatar
0 votes
0 answers
53 views

Does DPO update the weights in the same way that LORA/Fine-Tuining does?

I've been working on making a conversational customer service fine-tune for the past couple months and now I am looking to improve its failure recovery. For example, If the agent makes a mistake, how ...
the-test-set-is-all-you-need's user avatar
1 vote
0 answers
45 views

Why do mix models work?

Is there research on why models mixes work? One would expect that averaging the weights of two models would produce garbage, but many models mixes created by amateurs show that they not only work, but ...
allo's user avatar
  • 310
1 vote
3 answers
383 views

How to force LLM (like OpenAI ChatGPT) to output a variable list of values? [closed]

What prompt (or other technique) should I use with an LLM so that The result is guaranteed to be reliably parseable as a list of values (e.g. a Python list of strings) LLM would understand that a ...
Elijas Dapšauskas's user avatar
0 votes
1 answer
494 views

Getting started with training local LLM using python [closed]

As I'm completely new to this field, I find it hard to get started given the requirements I have. I'm a bit overwhelmed by all the models and options that are available. Even though it wasn't ...
Jeanluca Scaljeri's user avatar
0 votes
0 answers
33 views

Fetch latest data from Gemini model to build chat bot

I am building a chat bot based on gemini pro model through prompt engineering. The use case is simple. It allows user to ask a question about something specific which entails generating info about ...
Hackerz's user avatar
0 votes
0 answers
40 views

understanding the distribution shift problem in direct preference optimization (DPO)

I'm having trouble understanding this paragraph of the DPO paper: Why does it matter so much that the preference data distribution aligns with the reference model output distribution? My ...
Ivy Cao's user avatar
0 votes
0 answers
17 views

What reference documents exist for LLM inference engines and models?

For example, vllm is an inference engine, and according to their roadmap they will incororate vllm into several LLM API engines such as openllm, rayserve, and nvidia triton. What are examples of ...
Mark Harrison's user avatar
0 votes
0 answers
173 views

Where do you keep track of Chat History for an LLM Chat Application?

I am creating a LLM chat app with LangChain. I am keeping track of the chat history with a simple array that gets sent to the LLM with every prompt. I noticed LangChain also has classes that ...
Kokodoko's user avatar
  • 167
0 votes
0 answers
24 views

Chat with your DB: How to get AI to follow your rules?

Context/Background: I have a single table in MySQL with info on 'members', including things like location, favourite types of music, favourite camping spots, etc. Required Behaviour: I need the system ...
Blue Da Noob's user avatar
0 votes
0 answers
33 views

Any suggestions for transformer finetuning techniques ablation study?

I'm planning to fine tune a 7b parameter model for a research project. I understand the different steps of model fine tuning, namely Supervised fine tuning - where we train model on curated examples ...
kaiser's user avatar
  • 101
0 votes
1 answer
207 views

Are LLM parameters synonymous with dimensions?

For example, would a Large Language Model (LLM) with parameter size 140 Billion have 140 Billion dimensions as defined in deep learning as the number of nodes in the input layer? Another way to ask ...
geominded's user avatar
  • 101
0 votes
2 answers
313 views

In the paper "LLM in a flash," what is meant by an up projection or down projection layer?

In the paper, they first use the terms "up projection layer," and similarly for down projection, in this paragraph in the introduction: Row-column bundling: We store a concatenated row and ...
Tyler's user avatar
  • 103
0 votes
0 answers
51 views

What are some tips of curating a dataset to fine-tune a code-completion LLM?

There is a new SDK that I am working on and I want to know what are some ways of automatically curating a dataset to train a code-completing LLM to deploy as a VSCode plugin? Hacky ways are ...
Levent Ozbek's user avatar
0 votes
1 answer
50 views

how can I interpret attention weights matrix? Are they reliable?

I've fine-tuned two different models (Bert and Roberta) on a dataset for a binary classification task and I'm comparing the sentences where the models predict wrong. I decided to use attention weights ...
Shayan's user avatar
  • 21
1 vote
1 answer
31 views

publically available language models that can be used to train arbitrary language data?

I have sentence data in a language that is not widely in use and as such popular LLMs do not support the language. I want to train some language model such that given some question, it is able to ...
Neijal Kanderbalt's user avatar
0 votes
0 answers
14 views

Are There Scientific Papers on Methods for Robustifying LLMs and Current Challenges They Face?

I'm currently researching Large Language Models (LLMs) and am particularly interested in the recent advancements and challenges in this field. My focus is on understanding the methods being developed ...
Iman Mohammadi's user avatar
1 vote
2 answers
236 views

Which techniques are best suitable for explainable AI for LLM models

I am currently working with large language models like llama and mistral, interested in techniques for making these models more explainable. I am looking for some tools or techniques which can help me ...
Hiren Namera's user avatar
2 votes
0 answers
36 views

How to learn text style in an article using LLMs?

What is the best way to learn text style in an article? By text style I mean special characteristics and patterns inherent to different authors/group's writing style. For-example, author attribution ...
Shayan's user avatar
  • 21
1 vote
0 answers
17 views

Neural Machine Translation with multi-language input to a single-language output?

I'm looking for NMT paradigms where the input to the model is the same text in N languages (e.g., L1, L2, L3) and the output is the translation in a different target language (e.g., L4). However, I ...
yigitcankaya's user avatar