Skip to main content

Questions tagged [embeddings]

For questions about embeddings (not necessarily just word embeddings, for which there is a specific tag) in the context of machine learning.

Filter by
Sorted by
Tagged with
0 votes
0 answers
14 views

How to add/embed categorical features in transformers network?

I would like to give more context to my transformers by adding some metadata related to each token. This metadata is mostly categorical (3 fields, with 3 possible values for each field). In addition ...
JulienG's user avatar
0 votes
2 answers
66 views

How do Transformer models ensure unique token representations when combining embeddings and positional encodings?

In Transformer models, token embeddings are combined with positional encodings through element-wise addition to incorporate positional information. However, this raises a concern about the potential ...
The Pointer's user avatar
0 votes
0 answers
27 views

Skip-Gram Model description in word2vec explanation article

In his article word2vec Parameter Learning Explained Xin Rong says (page 7): Each output is computed using the same hidden->output matrix: $$ p(w_{c,j} = w_{O.c}|w_I)=y_{c,j}=\frac{exp(u_{c,j})}{\...
Damir Tenishev's user avatar
0 votes
1 answer
41 views

Dot notation for matrix element index in formula of word2vec embedding calculation

In his article word2vec Parameter Learning Explained Xin Rong uses the dot notation as an index for matrix (at the second page in the first formula). $$ h = W^{T}x = W^{T}_{(k,\cdot)} := {v^{T}}_{w_I} ...
Damir Tenishev's user avatar
0 votes
0 answers
13 views

best approach for training a model to determine patent prior art

I am looking to train a model that will take in text for a patent and be able to output the ids of patents that are most likely to be prior art for that idea. There is a ton of training data for this ...
Alexander Halpern's user avatar
2 votes
3 answers
117 views

How is it possible that RAG embeddings implemented with a different embedding model than the pre-trained LLM can work?

I have read and seen that you can use a different embeddings model for RAG embedding vectors than was used to train the original RAG. I can't get my head around how that can work! How is it possible ...
Mark_Sagecy's user avatar
1 vote
0 answers
17 views

Can embedding be tackled efficiently using MapReduce?

Embedding is necessary to convert text in vectors that are more tractable by machine learning models. Can embedding be carried out (efficiently) using the MapReduce approach? I suspect so: projecting ...
A. Darwin's user avatar
  • 143
0 votes
0 answers
67 views

Back-propagation through embedding layer

I was trying to understand how back-propogation works through embedding layer. After I wrote the equation, it seems like the gradient with respect to the weight of embedding layer is always 1. $$ \...
Irshad Basheer's user avatar
1 vote
0 answers
45 views

Obtaining canonical example sentences from an embedding

How do I obtain the best representative sentence for a single sentence embedding vector? I have a corpus of 30k articles, for which I have use the OpenAI API to find embeddings using ...
Thomas Browne's user avatar
0 votes
0 answers
16 views

Can 3D convolutions appropriately capture a frozen embedding space?

My project is a strange combination of NLP and Computer Vision. I have datapoints of 3D tensor where each element is a token in an NLP vocabulary. The vocabulary is around 1000 unique "words"...
schmixi's user avatar
0 votes
0 answers
10 views

How to quantify the impact of tokenizer output changes on text encoder embeddings?

In natural language processing, generating text embeddings involves two key modules: the tokenizer and the text encoder. The tokenizer converts text into a token vector, which the encoder then uses to ...
hanugm's user avatar
  • 3,930
0 votes
0 answers
15 views

How do I obtain human-readable descriptions (keywords, phrases, etc.) for an embedding vector

I am using an embedding model to obtain vectors corresponding to a group of short documents. I then cluster related documents using these vectors. For each cluster, I would like to describe to the ...
davidconner's user avatar
0 votes
1 answer
19 views

Best way to create a summary of variable length set of vectors where order does not matter

I'm trying to design a system to optimize over a variable-length set (like a sentence) of variable length vectors (like words). But unlike a sentence, the order of words does not matter. I'll have to ...
hasdrubal's user avatar
  • 101
0 votes
0 answers
32 views

Fine-tuned Stable Diffusion not converging with custom encoder

I'm currently fine-tuning a stable diffusion model for the task of dataset augmentation. I am training the model on 80k images from hte CelebA-attributes dataset, replacing the text encoder with ...
Tomas Premoli Muniagurria's user avatar
0 votes
1 answer
105 views

How do I code so that the embedding output and input share the same weight matrices?

I am trying to implement the Attention is All You Need paper from scratch. The authors mentioned in section 3.4 that "In our model, we share the same weight matrix between the two embedding ...
OneMoreGamble's user avatar
0 votes
0 answers
46 views

How to classify a new node into an existing cluster of nodes?

I have a graph with many disjoint subgraphs that are not connected to each other. Essentially these subgraphs could represent different clusters. What is a general process to figure out node ...
Kiran Manicka's user avatar
0 votes
1 answer
29 views

Navigating encoder-only text/sentence embedding models

tldr; I am researching/looking for vec2text methods that work (a) without training a special model to guide the mutations, and (b) without access to the model weights. I am doing some experiments in ...
Realz Slaw's user avatar
0 votes
0 answers
22 views

which the effective ranks of embedding matrices in Large LMs?

The hidden size of current LLMs is well in the 8K and above, and I was wondering, has the rank of the embedding matrix, and the rank of the transpose of the output matrix, been increasing with the ...
arivero's user avatar
  • 51
1 vote
1 answer
76 views

Approaching construction of model that interprets financial reports [closed]

I want to train a model to be able to interpret financial reports (from a company). Basically, I want to be able to extract relevant information without needing to read through hundreds of pages of ...
Hlkwtz's user avatar
  • 11
1 vote
2 answers
2k views

Do diffusion models take a long time to train?

I am trying to train a diffusion model (from scratch in pytorch). UNet used is not anything too fancy, takes in images and time step as input for about 512 time steps. I am using learnable embeddings ...
Aditya 's user avatar
0 votes
0 answers
28 views

Encoding categorical data with "many" unique values in neural network

I am new to machine learning, in fact, I am implementing my first deep neural network from scratch without any framework. The dataset has 3500 rows, and 4 categorial columns of which two have about ...
M a m a D's user avatar
  • 113
0 votes
2 answers
560 views

From where do the Encoders in Transformers gets Input Embedding from?

In Transformers Encoders, from where do the Encoders get Input Embedding from? So when a sentence is given to a transformer-based model it first tokenises the sentence and each token is mapped with ...
Swastik's user avatar
  • 101
0 votes
0 answers
14 views

Why is $z$ considered a random variable in the following formulation of Locally Linear Embedding Problem?

The following is the derivation of "Locally Linear Embedding Problem" from the book Machine Learning, 4-th edt, page 152, by E.Alpaydin. Why is $z$ considered a random variable so that $E(z)...
DSPinfinity's user avatar
1 vote
0 answers
231 views

Diffusion Model Failing to Learn

I'm trying to train a diffusion model to map between paired embedding spaces - ie using a CLIP text embedding to predict a CLIP image embedding. I have a working baseline model that predicts the ...
Karl's user avatar
  • 206
0 votes
0 answers
1k views

Why are the vector stores I generate via langchain FAISS from_documents always ~4.8 GB in size?

I am working on a small application to query a set of PDF documents. Specifically, I am aiming to provide the same query to each document in parallel, not to all documents as a whole. I have utilized <...
KDecker's user avatar
  • 101
2 votes
2 answers
3k views

How can I teach a book to an LLM?

I am trying to find out how I can teach the content of a whole, multiple hundert pages book to an LLM so that it "knows" all details and can be queried, give summaries etc. The book is one ...
dschuld's user avatar
  • 21
0 votes
0 answers
150 views

How to recover the text of the sentence from an embedding (vec2text)?

I'm trying to re-invent the code that would do it in the same way as in the vec2text article. Does anybody here have something similar?
Alex Fedotov's user avatar
0 votes
1 answer
369 views

Understanding embedding outputs in transformer models like CLIP

I'm working with OpenAI's CLIP model and trying to understand the output of the text encoder. When I input a short prompt like "cat", the output is a tensor of shape [77, 1024]. My ...
hanugm's user avatar
  • 3,930
0 votes
1 answer
29 views

Seq2Seq model- Confusing about the dimension of Seq2Seq model [closed]

I am new to Seq2Seq and hope to find a proper guildances, advices. I am doing a Project from an online course so I can not give the material but I got my Project notebook on Github I want to ask ...
QH.Chu's user avatar
  • 3
1 vote
1 answer
340 views

Can transformers autoregressively generate a sequence of embeddings (instead of predictions)?

Is it theoretically possible to use a transformer architecture to autoregressively generate a sequence of embedding vectors, instead of discrete tokens? For example, if I were to provide an input of a ...
Theo Coombes's user avatar
3 votes
1 answer
397 views

Match two paragraphs of text

I'm building a friend finder app and I need to match people based on a paragraph of text. Here is an example of what I mean: Person A: I love walking and going to the beach, I also love reading and ...
Dom's user avatar
  • 31
0 votes
1 answer
2k views

Why does LLM inference cost scale in both input tokens and output tokens?

EDIT This question was flawed. See my answer with help from commenters. Original question This question has been asked in other forums [1] [2] but I'm not sure I understand the claims, which are (...
llllvvuu's user avatar
  • 146
1 vote
0 answers
171 views

Is there an ANN vector-search index that supports incremental ingestion and deletion of elements?

I have looked at a few libraries for ANN search of high dimensional vectors. Although impressive, they come with a huge baggage of fine print. Many of them only support ingesting the vectors in one ...
morpheus's user avatar
  • 294
0 votes
1 answer
34 views

Ignoring aspects of text embeddings, e.g. making the embedding topic-agnostic

Imagine a large set of text embeddings (e.g. by OpenAI model), created on user inputs in a natural language interface (e.g. a semantic search app), which we want to cluster on some "non-topic ...
curl-up's user avatar
2 votes
0 answers
316 views

How is OpenAI embeddings obtained

There is OpenAI embedding API https://platform.openai.com/docs/guides/embeddings. How is this embedding related to the GPT3.5 transformer model architecture? Is it the vectors learned from the input ...
Salty Gold Fish's user avatar
0 votes
1 answer
23 views

How to learn Categorial Embeddings in Unsupervised Learning?

I want to cluster mixed-type tabular data, for the categorial columns I want to use Categorial Embeddings and then an Autoencoder Network before clustering with KMeans or similar. Now, when I want to ...
Jaanis's user avatar
  • 1
3 votes
0 answers
523 views

How to work with multiple embeddings?

This is a conceptual gap that I have concerning embeddings, and would really appreciate some help closing it. I understand when you embed a corpus for, let's say, a question-and-answer task you can ...
Ian Murray's user avatar
0 votes
2 answers
1k views

How do I choose a good treshold for classification (using cosine similarity scores)?

I am using openai's text-embedding-ada-002 embeddings model to do a semantic search on a database of articles to find articles that are most related to a given ...
Stefan's user avatar
  • 1
2 votes
0 answers
132 views

Nearest neighbour search in high dimension retrieves certain points too often

We represent some catalogue items (documents, music tracks, videos, whatever) as vectors embedded in R^d and use them to retrieve nearest neighbours to users query. The typical scenario is that users ...
Peter Franek's user avatar
1 vote
2 answers
955 views

How can I send vectors as a chat context?

Since the context/memory of a chat or question for LLMs more precisely GPT is limited to a token length I struggle about how to provide own data that the model got not trained on. A very common ...
dc10's user avatar
  • 111
0 votes
1 answer
161 views

What do different embedding functions do?

Embeddings are just arrays of numbers that represent some data (sentences or images). How do embedding functions differ. For example what advantage does OpenAI's embeddings API have over others?
felipe57387's user avatar
1 vote
0 answers
140 views

Is the input embedding split along the embedding dimension so that every head of the multi-head-attention module just gets a part of the input data?

So I found two contradictory explanations of the MHA (multi-head-self-attention-module): In the first approach, the input embedding (= the input matrix) is split along the embedding dimension and all ...
AAT's user avatar
  • 11
2 votes
1 answer
773 views

Creating a support chat bot for my business

I am trying to create a kind of support bot to answer questions from my clients about specific technical details about WordPress plugins that I sell. The goal is that the ...
digitalzoomstudio's user avatar
0 votes
0 answers
186 views

Using a VQ-VAE encoder's output to condition another model

I've trained a VQ-VAE on images that I want to use to condition another model (latent diffusion). The shape of my encoder's output is (4, 256, 56, 56) and I'm getting it using: ...
jbm's user avatar
  • 101
0 votes
1 answer
196 views

Are there methods which represent entire knowledge graphs in one embedding vector?

In a knowledge graph, embedding vectors can be learned for nodes (node embedding) and edges (edge embeddings). Is there a method to learn one single embedding vector for the entire knowledge graph?
skumaravel's user avatar
1 vote
1 answer
1k views

How to get meaningful vector embeddings for (lat, long) points and also GPS trajectories?

I have a data that consists of approx. 1.5M taxi trips in Porto, Portugal. (from: https://www.kaggle.com/competitions/pkdd-15-taxi-trip-time-prediction-ii/overview) Each of these trips have it's GPS ...
I am not a robot's user avatar
1 vote
0 answers
103 views

Privacy implications of storing and transmitting GPT-3 embeddings?

We are exploring implementing a feature where a user might enter "which product has optional all wheel drive" into a search input, which would be transformed to GPT-3 embeddings, and ...
jahghAAB's user avatar
1 vote
0 answers
181 views

Use multiple embeddings with the Transformer architecture

In their article about their system "MuseNet" OpenAI state the following: Embeddings We added several different kinds of embeddings to give the model more structural context. In addition to ...
Felix Schön's user avatar
1 vote
1 answer
82 views

How to embed quantitative variables?

I have 3 data types that I want to feed into a neural net. One, is a time series, which I am going to feed into the neural net directly. The second, are categorical variables that I am going to embed ...
Mattpats's user avatar
  • 113
3 votes
3 answers
3k views

How embeddings learned from one model can be used in another?

In the website the following explanation is provided about Embedding layer: The Embedding layer is initialized with random weights and will learn an embedding for all of the words in the training ...
Oculu 's user avatar
  • 43