Skip to main content
OverflowAI is here! AI power for your Stack Overflow for Teams knowledge community. Learn more

New answers tagged

0 votes
Accepted

Reference request: data efficiency of LLM pre-training

This article, cited on Reddit, provides an answer: Frank, Michael C. "Bridging the data gap between children and large language models." Trends in Cognitive Sciences (2023).: GPT-3 was ...
Jake Levi's user avatar
  • 101
0 votes

How can I get started with LLMs?

As a first-year AI student tackling Andrew Ng's course, you're on the right track! To enter into the exciting world of Large Language Models (LLMs) and generative AI development, here's are some steps ...
David Miller's user avatar
0 votes
Accepted

What should be Relationship between embedding dimension and context length?

Embedding Dimension (d_model): One word/token will be represented as "d_model" different numebrs. Larger the d_model,higher thwe capacityof storing the true meaning of the word It also ...
Kulin Patel's user avatar
0 votes

Can attention models be replaced by non-sigmoid activation functions?

The attention module can be seen in many way. Indeed, you can just see as a way to construct a set of weights on the fly (in poor words this is what it does): $$ A = softmax(QK/\sqrt{d})\\ output = A\...
Alberto's user avatar
  • 2,133
0 votes

What is an "inference kernel"?

I spoke briefly with ChatGPT on the matter; 3.5, I don't have access to 4.0. It says that it's not anything that it has or uses, so this is post-2022, so it's not anything that has to do with prompt ...
NinjaDarth's user avatar
4 votes
Accepted

Is there a relationship between tokens and parameters in LLMs?

Transformers parameters count is invariant to the context window (which by definition can be infinite, though the $O(n^2)$ complexity might hurt) Consider that the context window that you see in the ...
Alberto's user avatar
  • 2,133
0 votes

How to create 1 embedding for text + image

Translate the image into a description. Add the description of the image with the text. Embed both of those so you have the information from the image and the information from the text. How does this ...
Joseph's user avatar
  • 51

Top 50 recent answers are included