Questions tagged [embeddings]

For questions about embeddings (not necessarily just word embeddings, for which there is a specific tag) in the context of machine learning.

Filter by
Sorted by
Tagged with
0 votes
0 answers
16 views

Improving embedding similarity search of aggregated embeddings

I am building an author suggestion tool that proposes authors writing about similar topics as a given input text. I want to use embeddings for this. The way I currently do it is to store embeddings of ...
Stefan's user avatar
  • 1
0 votes
2 answers
42 views

How do I choose a good treshold for classification (using cosine similarity scores)?

I am using openai's text-embedding-ada-002 embeddings model to do a semantic search on a database of articles to find articles that are most related to a given ...
Stefan's user avatar
  • 1
2 votes
0 answers
88 views

Nearest neighbour search in high dimension retrieves certain points too often

We represent some catalogue items (documents, music tracks, videos, whatever) as vectors embedded in R^d and use them to retrieve nearest neighbours to users query. The typical scenario is that users ...
Peter Franek's user avatar
0 votes
0 answers
86 views

How can I send vectors as a chat context?

Since the context/memory of a chat or question for LLMs more precisely GPT is limited to a token length I struggle about how to provide own data that the model got not trained on. A very common ...
dc10's user avatar
  • 101
0 votes
0 answers
10 views

How to inspect a dimension in SBERT embeddings

I am using SBERT transformers to vectorize text. I want to use the sentence embeddings in a linear regression to predict a value or a classification. Some dimensions are highly colinear and result in ...
miguelmorin's user avatar
0 votes
1 answer
42 views

What do different embedding functions do?

Embeddings are just arrays of numbers that represent some data (sentences or images). How do embedding functions differ. For example what advantage does OpenAI's embeddings API have over others?
felipe57387's user avatar
0 votes
0 answers
12 views

How to perform graph embedding on a DAG graph with point weights and restore the resulting embedding vector to the DAG with point weights?

How to perform graph embedding on a directed acyclic graph with point weights and restore the resulting embedding vector to the directed acyclic graph with point weights?
user68072's user avatar
1 vote
0 answers
27 views

Is the input embedding split along the embedding dimension so that every head of the multi-head-attention module just gets a part of the input data?

So I found two contradictory explanations of the MHA (multi-head-self-attention-module): In the first approach, the input embedding (= the input matrix) is split along the embedding dimension and all ...
AAT's user avatar
  • 11
1 vote
1 answer
242 views

Creating a support chat bot for my business

I am trying to create a kind of Support Bot to answer my clients about specific technical details about WordPress plugins that I sell. The goal is that the /completitions api would be feeded a prompt ...
digitalzoomstudio's user avatar
0 votes
0 answers
14 views

How does FaceNet (or similar) bootstrap new faces?

In a metric learning system the system can be trained on known examples such that common classes (faces) are clustered together and separated from each other as much as possible. If triplet loss is ...
Mastiff's user avatar
  • 121
0 votes
0 answers
16 views

Does attention in transformers encode any information from positional embeddings?

I know we account for positional embeddings before feeding into attention layers, but would we be able to say that the Q and K dot products intrinsically encode relative positions
Manny's user avatar
  • 1
0 votes
0 answers
23 views

How to evaluate output of text generation models?

Evaluation of a wide variety of natural language generation (NLG) tasks is difficult. For instance, for a question answering model, it is hard for a human to quantify how well the model has answered a ...
Greggs's user avatar
  • 101
0 votes
0 answers
49 views

How do you specify the dimension to search for similarity in CLIP image embeddings?

I have a question about CLIP semantic image search. When you have an image of a person e.g. a skinny person wearing red shirt, clip will search for you similarity in all dimensions including body ...
off99555's user avatar
  • 325
0 votes
0 answers
49 views

Using a VQ-VAE encoder's output to condition another model

I've trained a VQ-VAE on images that I want to use to condition another model (latent diffusion). The shape of my encoder's output is (4, 256, 56, 56) and I'm getting it using: ...
jbm's user avatar
  • 101
0 votes
0 answers
14 views

Is it a problem if the target network and main network are the same?

I am currently working with this code that I found on Github: this is a DQN with Graph Neural Network (GNN) on TSP problem. So, in this architecture, the target network and the main network are the ...
elizabeth's user avatar
0 votes
0 answers
10 views

How to embed ordered integer values in a hidden layer

I want to embed integer positive values (ranging from 1 to 10000), which are ordered and not categorical, i.e., 1 is near to 2 than to 1000. This representation should be combined with numerical ...
Philipp123's user avatar
0 votes
0 answers
107 views

How do transformers handle multidimensional input?

Transformers work with lists of vectors, i.e. sentence of length SEQ_LEN, with each word having size EMBEDDING_DIM. Now, since the model still makes use of Dense layers internally, i.e. as in https://...
Daniel von Eschwege's user avatar
0 votes
1 answer
104 views

Are there methods which represent entire knowledge graphs in one embedding vector?

In a knowledge graph, embedding vectors can be learned for nodes (node embedding) and edges (edge embeddings). Is there a method to learn one single embedding vector for the entire knowledge graph?
skumaravel's user avatar
0 votes
0 answers
18 views

What's a good regression algorithm for handling tabular data that have categorical data, "list of words"

Problem statement: I want to predict future prices of trips based on historical pricing data. I'm looking for an algorithm that has the following features: Unsupervised algorithm Limit the amount of ...
zzzz8888's user avatar
0 votes
1 answer
208 views

How to get meaningful vector embeddings for (lat, long) points and also GPS trajectories?

I have a data that consists of approx. 1.5M taxi trips in Porto, Portugal. (from: https://www.kaggle.com/competitions/pkdd-15-taxi-trip-time-prediction-ii/overview) Each of these trips have it's GPS ...
I am not a robot's user avatar
0 votes
0 answers
57 views

Embedding layers/entities in openAI's Hide and seek paper

I've recently come across a youtube video about openAI's hide and seek paper (https://openai.com/blog/emergent-tool-use/) and got really fascinated about the paper itself. But as I digging in the ...
Hung Quang's user avatar
0 votes
0 answers
4 views

Are there standard token vocabularies for text encoding?

Are there good standard token vocabularies for text encoding tasks? I am looking for a set of syllables and letters, and maybe some very common words, in the best case sorted by frequency. If possible,...
allo's user avatar
  • 260
1 vote
0 answers
79 views

Privacy implications of storing and transmitting GPT-3 embeddings?

We are exploring implementing a feature where a user might enter "which product has optional all wheel drive" into a search input, which would be transformed to GPT-3 embeddings, and ...
jahghAAB's user avatar
1 vote
0 answers
52 views

Use multiple embeddings with the Transformer architecture

In their article about their system "MuseNet" OpenAI state the following: Embeddings We added several different kinds of embeddings to give the model more structural context. In addition to ...
Felix Schön's user avatar
0 votes
0 answers
53 views

How to evaluate the embeddings of a model?

If you have a task of extracting embeddings from a model (such as penultimate layer - pre-last layer of the model), would you train the model on a benchmark similar dataset (if there were) or train on ...
ali shibli's user avatar
1 vote
1 answer
66 views

How to embed quantitative variables?

I have 3 data types that I want to feed into a neural net. One, is a time series, which I am going to feed into the neural net directly. The second, are categorical variables that I am going to embed ...
Mattpats's user avatar
  • 113
0 votes
0 answers
14 views

Pose Sequence embedding

I am using a pose detection software which returns pose — a set of coordinates of thirty-ish points. I record a person for some duration (10 seconds to 1 minute). I want to have the embedding of the ...
khurshid juraev's user avatar
2 votes
1 answer
445 views

How embeddings learned from one model can be used in another?

In the website the following explanation is provided about Embedding layer: The Embedding layer is initialized with random weights and will learn an embedding for all of the words in the training ...
Oculu 's user avatar
  • 33
0 votes
0 answers
58 views

How to fix the embedding gauge in a BERT-like model?

I have a pre-trained BERT model from Huggingface, which I tune to categorize short texts (like tweets or slightly longer) into several thousand categories using triplet loss. As I understand, if I ...
Gena Kukartsev's user avatar
0 votes
0 answers
39 views

Can Graph Neural network leverage only the topological structure?

Graph Neural Networks (GNNs) are a powerful tool that allow learning on graphs by leveraging both the topological structure and the feature information for each node. For the particular problem I am ...
Marouane Jaakik's user avatar
2 votes
2 answers
952 views

What is the difference between representation and embedding?

As I searched about this two terms, I found they are somehow like each other, both try to create a vector from raw data as I understood. But, what is the difference of this two term?
aliiiiiiiiiiiiiiiiiiiii's user avatar
0 votes
1 answer
150 views

What is the distribution of autoencoder embeddings?

Is there any result on the distribution of autoencoder embeddings? For example, the following image (taken from this article) visualizes the latent space with t-SNE. As you can see, images from the ...
nalzok's user avatar
  • 211
1 vote
2 answers
2k views

How does GPT use the same embedding matrix for both input and output?

My understanding is that GPT uses the same embedding matrix for both inputs and output: Let $V$ be the vocab size, $D$ the number of embedding dimensions, and $E$ be a $V \times D$ embedding matrix: ...
SRobertJames's user avatar
1 vote
1 answer
377 views

What is a "continuous vector"?

I have seen the concept of a "continuous vector" described in the context of embeddings. For instance, this answer to a question on embeddings in the context of deep learning. I obviously ...
The Pointer's user avatar
0 votes
0 answers
21 views

Prototypical Network - Should I train my backbone or a separate embedder?

When I read the prototypical paper Prototypical Networks for Few-shot Learning, I understand in Eq. 1 that I should train $f_\phi$, which takes as input $x_i$, which is already an embedding of an ...
Adrien Nivaggioli's user avatar
7 votes
2 answers
1k views

Why are embeddings added, not concatenated?

Let's consider the following example from BERT I cannot understand why "the input embeddings are the sum of the token embeddings, the segmentation embeddings, and the position embeddings". ...
nalzok's user avatar
  • 211
0 votes
1 answer
38 views

Can I use Sentence-Bert to embed event triples?

I extracted event triples from sentences using OpenIE. Can I concatenate the components in the event triple to make it a sentence and use Sentence-Bert to embed the event? It seems no one has done ...
user900476's user avatar
0 votes
0 answers
41 views

Could I cluster the audio clips in order to improve the speed of their classification?

I have a neural network which is very resource intensive and is used to classify audio clips. The classification is done in batches, where I record for a set period of time and then go through and ...
GILO's user avatar
  • 101
1 vote
1 answer
483 views

Is "node embedding" in GNN analogous to "hidden layer" of FFN?

So in Graph Neural Network (GNN) we have node embeddings which is a feature vector that describes the node, is it analogous to hidden layer of Artificial neural network such as feed-forward neural ...
user0193's user avatar
  • 145
0 votes
1 answer
65 views

Sensible integer embedding/encoding for distinguishing elements of a set?

I am trying to train a model that takes in a set of feature vectors (which comes with an ID to uniquely identify elements of the set) and outputs a target for each element in the set (in a permutation-...
XYZT's user avatar
  • 101
1 vote
1 answer
962 views

Sequence Embedding using embedding layer: how does the network architecture influence it?

I want to obtain a dense vector representation of protein sequences so that I can meaningfully represent them in an embedding space. We can consider them as sequences of letters, in particular there ...
HelpNeederStudent's user avatar
0 votes
2 answers
127 views

How do multimodal models establish connections between different modes?

I am specifically interested in data2vec, Meta's new model that can convert image, text, and sound data into a unified neural network representation. To my understanding, they did this through self-...
user3576467's user avatar
1 vote
1 answer
223 views

Perform clustering on high dimensional data

Recently I trained a BYOL model on a set of images to learn an embedding space where similar vectors are close by. The performance was fantastic when I performed approximate K-nearest neighbours ...
VEDANT JOSHI's user avatar
3 votes
1 answer
138 views

What are knowledge graph embeddings?

What are knowledge graph embeddings? How are they useful? Are there any extensive reviews on the subject to know all the details? Note that I am asking this question just to give a quick overview of ...
nbro's user avatar
  • 38.2k
2 votes
0 answers
28 views

Is there any way to force one input have more effect on model?

Now I am working on building a deep learning model for a regression problem. I used 50 inputs and try to add one new categorical input. The problem is that this one input is much more important than ...
taegyun kim's user avatar
0 votes
1 answer
67 views

Best way to resize 3d to 2d matrix

I have a (5, 128, 768) matrix, that is, I have 5 embedding spaces of shape (128, 768). Since they all keep a relation, and for ...
user16696448's user avatar
1 vote
0 answers
31 views

How do the scale of an embedding affects a downstream task?

I am currently training a neural network in a self-supervised fashion, using Contrastive Loss and I want to use that network then to fine-tune it in a classification task with a small fraction of the ...
adam's user avatar
  • 11
1 vote
0 answers
24 views

What is the reason behind using node embeddings?

I was reading Chapter 3 from the following book (here) on graph representation learning. The chapter is about node embeddings. Question: What is the point of using node embeddings? Do we use them: to ...
Rocky the Owl's user avatar
1 vote
0 answers
24 views

Is graph embedding linear in its maintaining of graph geometry?

It is claimed that the main goal of graph embedding methods is to pack every node's properties into a vector with a smaller dimension, so node similarity in the original complex irregular spaces can ...
Nick's user avatar
  • 251
7 votes
3 answers
11k views

How to determine the embedding size?

When we are training a neural network, we are going to determine the embedding size to convert the categorical (in NLP, for instance) or continuous (in computer vision or voice) information to hidden ...
Lerner Zhang's user avatar