Questions tagged [embeddings]
For questions about embeddings (not necessarily just word embeddings, for which there is a specific tag) in the context of machine learning.
80
questions
0
votes
0
answers
22
views
Encoding categorical data with "many" unique values in neural network
I am new to machine learning, in fact, I am implementing my first deep neural network from scratch without any framework.
The dataset has 3500 rows, and 4 categorial columns of which two have about ...
- 113
0
votes
2
answers
202
views
From where do the Encoders in Transformers gets Input Embedding from?
In Transformers Encoders, from where do the Encoders get Input Embedding from?
So when a sentence is given to a transformer-based model it first tokenises the sentence and each token is mapped with ...
- 1
0
votes
0
answers
14
views
Why is $z$ considered a random variable in the following formulation of Locally Linear Embedding Problem?
The following is the derivation of "Locally Linear Embedding Problem" from the book
Machine Learning, 4-th edt, page 152, by E.Alpaydin.
Why is $z$ considered a random variable so that $E(z)...
- 492
1
vote
0
answers
73
views
Diffusion Model Failing to Learn
I'm trying to train a diffusion model to map between paired embedding spaces - ie using a CLIP text embedding to predict a CLIP image embedding. I have a working baseline model that predicts the ...
- 206
0
votes
0
answers
567
views
Why are the vector stores I generate via langchain FAISS from_documents always ~4.8 GB in size?
I am working on a small application to query a set of PDF documents. Specifically, I am aiming to provide the same query to each document in parallel, not to all documents as a whole.
I have utilized <...
- 101
0
votes
1
answer
250
views
How can I teach a book to an LLM?
I am trying to find out how I can teach the content of a whole, multiple hundert pages book to an LLM so that it "knows" all details and can be queried, give summaries etc. The book is one ...
- 1
0
votes
0
answers
69
views
How to recover the text of the sentence from an embedding (vec2text)?
I'm trying to re-invent the code that would do it in the same way as in the vec2text article. Does anybody here have something similar?
- 101
0
votes
1
answer
122
views
Understanding embedding outputs in transformer models like CLIP
I'm working with OpenAI's CLIP model and trying to understand the output of the text encoder. When I input a short prompt like "cat", the output is a tensor of shape [77, 1024]. My ...
- 3,612
0
votes
1
answer
21
views
Seq2Seq model- Confusing about the dimension of Seq2Seq model [closed]
I am new to Seq2Seq and hope to find a proper guildances, advices.
I am doing a Project from an online course so I can not give the material but I got my Project notebook on Github
I want to ask ...
- 3
1
vote
1
answer
123
views
Can transformers autoregressively generate a sequence of embeddings (instead of predictions)?
Is it theoretically possible to use a transformer architecture to autoregressively generate a sequence of embedding vectors, instead of discrete tokens?
For example, if I were to provide an input of a ...
- 11
3
votes
1
answer
146
views
Match two paragraphs of text
I'm building a friend finder app and I need to match people based on a paragraph of text. Here is an example of what I mean:
Person A:
I love walking and going to the beach, I also love reading and ...
- 31
0
votes
1
answer
574
views
Why does LLM inference cost scale in both input tokens and output tokens?
EDIT
This question was flawed. See my answer with help from commenters.
Original question
This question has been asked in other forums [1] [2] but I'm not sure I understand the claims, which are (...
- 146
0
votes
0
answers
79
views
Is there an ANN vector-search index that supports incremental ingestion and deletion of elements?
I have looked at a few libraries for ANN search of high dimensional vectors. Although impressive, they come with a huge baggage of fine print. Many of them only support ingesting the vectors in one ...
- 244
0
votes
0
answers
11
views
Update OpenAI embedding based on own domain corpus
I have a large domain corpus. Is there anyway to update the word/sentence/document embedding obtained from OpenAI embedding API based on my own domain corpus? There may be some words in my corpus that ...
- 101
0
votes
0
answers
7
views
Mining overlapping orthogonal clusters from high dimensional data
What are the recommended strategies for identifying overlapping and somewhat orthogonal clusters in a large, high-dimensional dataset?
As an illustration, consider a dataset comprised of various ...
- 1
0
votes
1
answer
28
views
Ignoring aspects of text embeddings, e.g. making the embedding topic-agnostic
Imagine a large set of text embeddings (e.g. by OpenAI model), created on user inputs in a natural language interface (e.g. a semantic search app), which we want to cluster on some "non-topic ...
- 1
0
votes
0
answers
178
views
How is OpenAI embeddings obtained
There is OpenAI embedding API https://platform.openai.com/docs/guides/embeddings. How is this embedding related to the GPT3.5 transformer model architecture? Is it the vectors learned from the input ...
- 101
0
votes
0
answers
97
views
Regarding the use of Time2Vec as positional encoding for Timeseries Transformer
Although the transformer architecture was originally designed for NLP, there exists several articles and papers that attempt to apply the same architecture for numerical timeseries classification. ...
- 21
0
votes
0
answers
26
views
Are the dimensions in embedding vectors ordered (similar to PCA)?
I am getting started with the vector embeddings. I have a general question about the embedding vectors generated by popular algorithms.
In PCA, usually, there is an implicit order of importance in the ...
- 101
0
votes
1
answer
22
views
How to learn Categorial Embeddings in Unsupervised Learning?
I want to cluster mixed-type tabular data, for the categorial columns I want to use Categorial Embeddings and then an Autoencoder Network before clustering with KMeans or similar.
Now, when I want to ...
- 1
0
votes
0
answers
65
views
Is it possible to generate new text matching a vector embedding with an LLM
I'm interested in generating variations of text using an LLM - is it possible to take a text embedding, move it in different directions in vector space, and generate new text from the resulting ...
- 101
1
vote
0
answers
185
views
How to work with multiple embeddings?
This is a conceptual gap that I have concerning embeddings, and would really appreciate some help closing it.
I understand when you embed a corpus for, let's say, a question-and-answer task you can ...
- 11
0
votes
0
answers
114
views
Improving embedding similarity search of aggregated embeddings
I am building an author suggestion tool that proposes authors writing about similar topics as a given input text. I want to use embeddings for this. The way I currently do it is to store embeddings of ...
- 1
0
votes
2
answers
528
views
How do I choose a good treshold for classification (using cosine similarity scores)?
I am using openai's text-embedding-ada-002 embeddings model to do a semantic search on a database of articles to find articles that are most related to a given ...
- 1
2
votes
0
answers
116
views
Nearest neighbour search in high dimension retrieves certain points too often
We represent some catalogue items (documents, music tracks, videos, whatever) as vectors embedded in R^d and use them to retrieve nearest neighbours to users query. The typical scenario is that users ...
- 432
0
votes
2
answers
589
views
How can I send vectors as a chat context?
Since the context/memory of a chat or question for LLMs more precisely GPT is limited to a token length I struggle about how to provide own data that the model got not trained on.
A very common ...
- 101
0
votes
0
answers
23
views
How to inspect a dimension in SBERT embeddings
I am using SBERT transformers to vectorize text. I want to use the sentence embeddings in a linear regression to predict a value or a classification. Some dimensions are highly colinear and result in ...
- 17
0
votes
1
answer
88
views
What do different embedding functions do?
Embeddings are just arrays of numbers that represent some data (sentences or images). How do embedding functions differ. For example what advantage does OpenAI's embeddings API have over others?
0
votes
0
answers
29
views
How to perform graph embedding on a DAG graph with point weights and restore the resulting embedding vector to the DAG with point weights?
How to perform graph embedding on a directed acyclic graph with point weights and restore the resulting embedding vector to the directed acyclic graph with point weights?
- 33
1
vote
0
answers
69
views
Is the input embedding split along the embedding dimension so that every head of the multi-head-attention module just gets a part of the input data?
So I found two contradictory explanations of the MHA (multi-head-self-attention-module):
In the first approach, the input embedding (= the input matrix) is split along the embedding dimension and all ...
- 11
2
votes
1
answer
480
views
Creating a support chat bot for my business
I am trying to create a kind of Support Bot to answer my clients about specific technical details about WordPress plugins that I sell.
The goal is that the /completitions api would be feeded a prompt ...
0
votes
0
answers
18
views
How does FaceNet (or similar) bootstrap new faces?
In a metric learning system the system can be trained on known examples such that common classes (faces) are clustered together and separated from each other as much as possible. If triplet loss is ...
- 121
0
votes
0
answers
18
views
Does attention in transformers encode any information from positional embeddings?
I know we account for positional embeddings before feeding into attention layers, but would we be able to say that the Q and K dot products intrinsically encode relative positions
- 1
0
votes
0
answers
62
views
How to evaluate output of text generation models?
Evaluation of a wide variety of natural language generation (NLG) tasks is difficult. For instance, for a question answering model, it is hard for a human to quantify how well the model has answered a ...
- 101
0
votes
0
answers
61
views
How do you specify the dimension to search for similarity in CLIP image embeddings?
I have a question about CLIP semantic image search. When you have an image of a person e.g. a skinny person wearing red shirt, clip will search for you similarity in all dimensions including body ...
- 325
0
votes
0
answers
104
views
Using a VQ-VAE encoder's output to condition another model
I've trained a VQ-VAE on images that I want to use to condition another model (latent diffusion). The shape of my encoder's output is (4, 256, 56, 56) and I'm getting it using:
...
- 101
0
votes
0
answers
22
views
Is it a problem if the target network and main network are the same?
I am currently working with this code that I found on Github: this is a DQN with Graph Neural Network (GNN) on TSP problem.
So, in this architecture, the target network and the main network are the ...
0
votes
0
answers
12
views
How to embed ordered integer values in a hidden layer
I want to embed integer positive values (ranging from 1 to 10000), which are ordered and not categorical, i.e., 1 is near to 2 than to 1000. This representation should be combined with numerical ...
- 61
0
votes
0
answers
235
views
How do transformers handle multidimensional input?
Transformers work with lists of vectors, i.e. sentence of length SEQ_LEN, with each word having size EMBEDDING_DIM. Now, since the model still makes use of Dense layers internally, i.e. as in https://...
0
votes
1
answer
166
views
Are there methods which represent entire knowledge graphs in one embedding vector?
In a knowledge graph, embedding vectors can be learned for nodes (node embedding) and edges (edge embeddings). Is there a method to learn one single embedding vector for the entire knowledge graph?
0
votes
0
answers
19
views
What's a good regression algorithm for handling tabular data that have categorical data, "list of words"
Problem statement: I want to predict future prices of trips based on historical pricing data.
I'm looking for an algorithm that has the following features:
Unsupervised algorithm
Limit the amount of ...
- 1
1
vote
1
answer
550
views
How to get meaningful vector embeddings for (lat, long) points and also GPS trajectories?
I have a data that consists of approx. 1.5M taxi trips in Porto, Portugal. (from: https://www.kaggle.com/competitions/pkdd-15-taxi-trip-time-prediction-ii/overview) Each of these trips have it's GPS ...
1
vote
0
answers
95
views
Privacy implications of storing and transmitting GPT-3 embeddings?
We are exploring implementing a feature where a user might enter "which product has optional all wheel drive" into a search input, which would be transformed to GPT-3 embeddings, and ...
- 11
1
vote
0
answers
116
views
Use multiple embeddings with the Transformer architecture
In their article about their system "MuseNet" OpenAI state the following:
Embeddings
We added several different kinds of embeddings to give the model more
structural context. In addition to ...
- 31
1
vote
1
answer
74
views
How to embed quantitative variables?
I have 3 data types that I want to feed into a neural net.
One, is a time series, which I am going to feed into the neural net directly. The second, are categorical variables that I am going to embed ...
- 113
3
votes
2
answers
1k
views
How embeddings learned from one model can be used in another?
In the website the following explanation is provided about Embedding layer:
The Embedding layer is initialized with random weights and will learn
an embedding for all of the words in the training ...
- 43
4
votes
2
answers
2k
views
What is the difference between representation and embedding?
As I searched about this two terms, I found they are somehow like each other, both try to create a vector from raw data as I understood. But, what is the difference of this two term?
0
votes
1
answer
207
views
What is the distribution of autoencoder embeddings?
Is there any result on the distribution of autoencoder embeddings?
For example, the following image (taken from this article) visualizes the latent space with t-SNE. As you can see, images from the ...
- 271
1
vote
2
answers
3k
views
How does GPT use the same embedding matrix for both input and output?
My understanding is that GPT uses the same embedding matrix for both inputs and output: Let $V$ be the vocab size, $D$ the number of embedding dimensions, and $E$ be a $V \times D$ embedding matrix:
...
- 63
2
votes
1
answer
601
views
What is a "continuous vector"?
I have seen the concept of a "continuous vector" described in the context of embeddings. For instance, this answer to a question on embeddings in the context of deep learning. I obviously ...
- 547