Questions tagged [embeddings]

For questions about embeddings (not necessarily just word embeddings, for which there is a specific tag) in the context of machine learning.

Filter by
Sorted by
Tagged with
0 votes
0 answers
4 views

Prototypical Network - Should I train my Backbone or a separate Embedder?

When I read the prototypical paper (https://arxiv.org/pdf/1703.05175.pdf), I understand in Eq.1 that I should train f_phi, which takes as input x_i, which is already an embedding of an image. So, I ...
user avatar
3 votes
1 answer
47 views

Why are embeddings added, not concatenated?

Let's consider the following example from BERT I cannot understand why "the input embeddings are the sum of the token embeddings, the segmentation embeddings, and the position embeddings". ...
user avatar
  • 161
0 votes
1 answer
20 views

Can I use Sentence-Bert to embed event triples?

I extracted event triples from sentences using OpenIE. Can I concatenate the components in the event triple to make it a sentence and use Sentence-Bert to embed the event? It seems no one has done ...
user avatar
0 votes
0 answers
21 views

Could I cluster the audio clips in order to improve the speed of their classification?

I have a neural network which is very resource intensive and is used to classify audio clips. The classification is done in batches, where I record for a set period of time and then go through and ...
user avatar
  • 101
0 votes
0 answers
14 views

Transformers for regression on permutation of fixed size sequence?

Transformers have shown remarkable performance operating on sequences, but are equivariant to the order in the input sequence. Positional Encoding alleviates that problem, but how good is it? In my ...
user avatar
  • 246
0 votes
0 answers
29 views

How to embed numerical inputs of unknown/varying size into some lower dimensional space?

I am interested in how one may handle numerical input data of unknown and varying input size for neural networks. Specifically, how would you approach embedding this data to some lower, constant ...
user avatar
1 vote
1 answer
83 views

Is "node embedding" in GNN analogous to "hidden layer" of FFN?

So in Graph Neural Network (GNN) we have node embeddings which is a feature vector that describes the node, is it analogous to hidden layer of Artificial neural network such as feed-forward neural ...
user avatar
  • 135
0 votes
0 answers
18 views

Variational AutoEncoders | Is Latent space an Embedding space? [duplicate]

I am learning about Variational Autoencoders and it is mentioned that the objective of an encoder is to produce a latent space, "encoding vector". Question: Is latent space just an "...
user avatar
  • 135
0 votes
1 answer
32 views

Sensible integer embedding/encoding for distinguishing elements of a set?

I am trying to train a model that takes in a set of feature vectors (which comes with an ID to uniquely identify elements of the set) and outputs a target for each element in the set (in a permutation-...
user avatar
  • 101
1 vote
1 answer
182 views

Sequence Embedding using embedding layer: how does the network architecture influence it?

I want to obtain a dense vector representation of protein sequences so that I can meaningfully represent them in an embedding space. We can consider them as sequences of letters, in particular there ...
user avatar
0 votes
2 answers
76 views

How do multimodal models establish connections between different modes?

I am specifically interested in data2vec, Meta's new model that can convert image, text, and sound data into a unified neural network representation. To my understanding, they did this through self-...
user avatar
1 vote
1 answer
67 views

Perform clustering on high dimensional data

Recently I trained a BYOL model on a set of images to learn an embedding space where similar vectors are close by. The performance was fantastic when I performed approximate K-nearest neighbours ...
user avatar
3 votes
1 answer
103 views

What are knowledge graph embeddings?

What are knowledge graph embeddings? How are they useful? Are there any extensive reviews on the subject to know all the details? Note that I am asking this question just to give a quick overview of ...
user avatar
  • 33.8k
2 votes
0 answers
27 views

Is there any way to force one input have more effect on model?

Now I am working on building a deep learning model for a regression problem. I used 50 inputs and try to add one new categorical input. The problem is that this one input is much more important than ...
user avatar
0 votes
1 answer
25 views

Best way to resize 3d to 2d matrix

I have a (5, 128, 768) matrix, that is, I have 5 embedding spaces of shape (128, 768). Since they all keep a relation, and for ...
user avatar
  • 111
1 vote
0 answers
19 views

How do the scale of an embedding affects a downstream task?

I am currently training a neural network in a self-supervised fashion, using Contrastive Loss and I want to use that network then to fine-tune it in a classification task with a small fraction of the ...
user avatar
  • 11
1 vote
0 answers
18 views

What is the reason behind using node embeddings?

I was reading Chapter 3 from the following book (here) on graph representation learning. The chapter is about node embeddings. Question: What is the point of using node embeddings? Do we use them: to ...
user avatar
0 votes
0 answers
12 views

Mapping ground truth to downsampled embeddings

I am currently pulling embeddings out of the mid layers of PSPNet. I was wondering if anyone knows of a way to see what pixels in the ground truth map to the pixels in the intermediate layers? e.g. we ...
user avatar
1 vote
0 answers
19 views

Is graph embedding linear in its maintaining of graph geometry?

It is claimed that the main goal of graph embedding methods is to pack every node's properties into a vector with a smaller dimension, so node similarity in the original complex irregular spaces can ...
user avatar
  • 251
0 votes
0 answers
22 views

Why use two different embeddings for actions in this paper?

I was reading this paper Top-𝐾 Off-Policy Correction for a REINFORCE Recommender System and I'm wondering is there a particular advantage to use different embeddings for actions, one embedding is ...
user avatar
  • 509
3 votes
2 answers
2k views

How to determine the embedding size?

When we are training a neural network, we are going to determine the embedding size to convert the categorical (in NLP, for instance) or continuous (in computer vision or voice) information to hidden ...
user avatar
1 vote
0 answers
28 views

Why are BERT embeddings interpreted as representations of the corresponding words?

It's often assumed in literature that BERT embeddings are contextual representations of the corresponding word. That is, if the 5th word is "cold", then the 5th BERT embedding is a ...
user avatar
2 votes
1 answer
81 views

What is an "input embedding" in the context of NLP?

When reading about NLP, I saw it said that "input embeddings" are a main element of encoder-decoder learning frameworks for sequence modelling. What is an "input embedding" in the ...
user avatar
5 votes
2 answers
327 views

Is an embedding a representation of a word or its meaning?

What does the term "embedding" actually mean? An embedding is a vector, but is that vector a representation of a word or its meaning? Literature loosely uses the word for both purposes. ...
user avatar
  • 3,099
2 votes
2 answers
147 views

Is it realistic to train a transformer-based model (e.g. GPT) in a self-supervised way directly on the Mel spectrogram?

In music information retrieval, one usually converts an audio signal into some kind "sequence of frequency-vectors", such as STFT or Mel-spectrogram. I'm wondering if it is a good idea to ...
user avatar
0 votes
1 answer
54 views

How does t-SNE preserves embedding orders?

According to the triplet loss Wikipedia page: t-SNE (t-distributed Stochastic Neighbor Embedding) preserves embedding orders via probability distributions, whereas triplet loss works directly on ...
user avatar
2 votes
0 answers
21 views

Embedding from Transformer-based model from paragraph or documnet (like Doc2Vec)

I have a set of data that contains the different lengths of sequences. On average the sequence length is 600. The dataset is like this: ...
user avatar
0 votes
1 answer
129 views

Are the Word2Vec encoded embeddings available online? [closed]

I am trying to do an NLP project and was wondering if there is anywhere online where the Word2Vec embeddings are stored (the actual n-dimmensional vectors). I want to search up a word and see what its ...
user avatar
0 votes
1 answer
40 views

Converting age and sex variables to a 64-unit dense layer

I am studying a preprint for my own learning (https://www.medrxiv.org/content/medrxiv/early/2020/04/27/2020.04.23.20067967.full.pdf) and I am befuddled by the following detail of the neural network ...
user avatar
3 votes
1 answer
197 views

How does the embeddings work in vision transformer from paper?

I get the part from the paper where the image is split into P say 16x16 (smaller images) patches and then you have to ...
user avatar
  • 233
1 vote
0 answers
24 views

Is there any research work that shows that we should explicitly mark the word boundaries for 1D CNNs?

I'm doing character embedding for NLP tasks using one-dimensional convolutional neural networks (see Chiu and Nichols (2016) for the motivation). I haven't found any empirical evidence of whether or ...
user avatar
2 votes
1 answer
98 views

In the machine learning literature, what does it mean to say that something is "embedded" in some space?

In the machine learning literature, I often see it said that something is "embedded" in some space. For instance, that something is "embedded" in feature space, or that our data ...
user avatar
1 vote
1 answer
163 views

What is the role of embeddings in a deep recurrent Q network?

When describing the model architecture for a deep recurrent q network, the authors of the paper Learning to Communicate with Deep Multi-Agent Reinforcement Learning each agent consists of a recurrent ...
user avatar
  • 43
23 votes
5 answers
10k views

What is the difference between latent and embedding spaces?

In general, the word "latent" means "hidden" and "to embed" means "to incorporate". In machine learning, the expressions "hidden (or latent) space" ...
user avatar
  • 33.8k
1 vote
1 answer
2k views

How is the constraint $\|f(x)\|_{2}=1$ enforced for the embedding $f(x)$ in the FaceNet paper?

In the FaceNet paper, under section 3.2, the authors mention that: The embedding is represented by $f(x) \in \mathbb{R}^{d}$. It embeds an image $x$ into a $d$-dimensional Euclidean space. ...
user avatar
4 votes
2 answers
140 views

What is the intuition behind how word embeddings bring information to a neural network?

How is it that a word embedding layer (say word2vec) brings more insights to the neural network compared to a simple one-hot encoded layer? I understand how the word embedding carries some semantic ...
user avatar
  • 189