Questions tagged [embeddings]
For questions about embeddings (not necessarily just word embeddings, for which there is a specific tag) in the context of machine learning.
36
questions
0
votes
0
answers
4
views
Prototypical Network - Should I train my Backbone or a separate Embedder?
When I read the prototypical paper (https://arxiv.org/pdf/1703.05175.pdf), I understand in Eq.1 that I should train f_phi, which takes as input x_i, which is already an embedding of an image.
So, I ...
3
votes
1
answer
47
views
Why are embeddings added, not concatenated?
Let's consider the following example from BERT
I cannot understand why "the input embeddings are the sum of the token embeddings, the segmentation embeddings, and the position embeddings". ...
- 161
0
votes
1
answer
20
views
Can I use Sentence-Bert to embed event triples?
I extracted event triples from sentences using OpenIE. Can I concatenate the components in the event triple to make it a sentence and use Sentence-Bert to embed the event?
It seems no one has done ...
- 21
0
votes
0
answers
21
views
Could I cluster the audio clips in order to improve the speed of their classification?
I have a neural network which is very resource intensive and is used to classify audio clips. The classification is done in batches, where I record for a set period of time and then go through and ...
- 101
0
votes
0
answers
14
views
Transformers for regression on permutation of fixed size sequence?
Transformers have shown remarkable performance operating on sequences, but are equivariant to the order in the input sequence. Positional Encoding alleviates that problem, but how good is it?
In my ...
- 246
0
votes
0
answers
29
views
How to embed numerical inputs of unknown/varying size into some lower dimensional space?
I am interested in how one may handle numerical input data of unknown and varying input size for neural networks.
Specifically, how would you approach embedding this data to some lower, constant ...
1
vote
1
answer
83
views
Is "node embedding" in GNN analogous to "hidden layer" of FFN?
So in Graph Neural Network (GNN) we have node embeddings which is a feature vector that describes the node, is it analogous to hidden layer of Artificial neural network such as feed-forward neural ...
- 135
0
votes
0
answers
18
views
Variational AutoEncoders | Is Latent space an Embedding space? [duplicate]
I am learning about Variational Autoencoders and it is mentioned that the objective of an encoder is to produce a latent space, "encoding vector".
Question: Is latent space just an "...
- 135
0
votes
1
answer
32
views
Sensible integer embedding/encoding for distinguishing elements of a set?
I am trying to train a model that takes in a set of feature vectors (which comes with an ID to uniquely identify elements of the set) and outputs a target for each element in the set (in a permutation-...
- 101
1
vote
1
answer
182
views
Sequence Embedding using embedding layer: how does the network architecture influence it?
I want to obtain a dense vector representation of protein sequences so that I can meaningfully represent them in an embedding space. We can consider them as sequences of letters, in particular there ...
0
votes
2
answers
76
views
How do multimodal models establish connections between different modes?
I am specifically interested in data2vec, Meta's new model that can convert image, text, and sound data into a unified neural network representation. To my understanding, they did this through self-...
- 101
1
vote
1
answer
67
views
Perform clustering on high dimensional data
Recently I trained a BYOL model on a set of images to learn an embedding space where similar vectors are close by. The performance was fantastic when I performed approximate K-nearest neighbours ...
- 21
3
votes
1
answer
103
views
What are knowledge graph embeddings?
What are knowledge graph embeddings? How are they useful? Are there any extensive reviews on the subject to know all the details? Note that I am asking this question just to give a quick overview of ...
- 33.8k
2
votes
0
answers
27
views
Is there any way to force one input have more effect on model?
Now I am working on building a deep learning model for a regression problem. I used 50 inputs and try to add one new categorical input. The problem is that this one input is much more important than ...
- 21
0
votes
1
answer
25
views
Best way to resize 3d to 2d matrix
I have a (5, 128, 768) matrix, that is, I have 5 embedding spaces of shape (128, 768). Since they all keep a relation, and for ...
- 111
1
vote
0
answers
19
views
How do the scale of an embedding affects a downstream task?
I am currently training a neural network in a self-supervised fashion, using Contrastive Loss and I want to use that network then to fine-tune it in a classification task with a small fraction of the ...
- 11
1
vote
0
answers
18
views
What is the reason behind using node embeddings?
I was reading Chapter 3 from the following book (here) on graph representation learning. The chapter is about node embeddings.
Question: What is the point of using node embeddings?
Do we use them:
to ...
0
votes
0
answers
12
views
Mapping ground truth to downsampled embeddings
I am currently pulling embeddings out of the mid layers of PSPNet. I was wondering if anyone knows of a way to see what pixels in the ground truth map to the pixels in the intermediate layers? e.g. we ...
1
vote
0
answers
19
views
Is graph embedding linear in its maintaining of graph geometry?
It is claimed that the main goal of graph embedding methods is to pack every node's properties into a vector with a smaller dimension, so node similarity in the original complex irregular spaces can ...
- 251
0
votes
0
answers
22
views
Why use two different embeddings for actions in this paper?
I was reading this paper Top-𝐾 Off-Policy Correction for a REINFORCE Recommender System and I'm wondering is there a particular advantage to use different embeddings for actions, one embedding is ...
- 509
3
votes
2
answers
2k
views
How to determine the embedding size?
When we are training a neural network, we are going to determine the embedding size to convert the categorical (in NLP, for instance) or continuous (in computer vision or voice) information to hidden ...
- 741
1
vote
0
answers
28
views
Why are BERT embeddings interpreted as representations of the corresponding words?
It's often assumed in literature that BERT embeddings are contextual representations of the corresponding word. That is, if the 5th word is "cold", then the 5th BERT embedding is a ...
2
votes
1
answer
81
views
What is an "input embedding" in the context of NLP?
When reading about NLP, I saw it said that "input embeddings" are a main element of encoder-decoder learning frameworks for sequence modelling. What is an "input embedding" in the ...
- 423
5
votes
2
answers
327
views
Is an embedding a representation of a word or its meaning?
What does the term "embedding" actually mean?
An embedding is a vector, but is that vector a representation of a word or its meaning? Literature loosely uses the word for both purposes. ...
- 3,099
2
votes
2
answers
147
views
Is it realistic to train a transformer-based model (e.g. GPT) in a self-supervised way directly on the Mel spectrogram?
In music information retrieval, one usually converts an audio signal into some kind "sequence of frequency-vectors", such as STFT or Mel-spectrogram.
I'm wondering if it is a good idea to ...
- 83
0
votes
1
answer
54
views
How does t-SNE preserves embedding orders?
According to the triplet loss Wikipedia page:
t-SNE (t-distributed Stochastic Neighbor Embedding) preserves embedding orders via probability distributions, whereas triplet loss works directly on ...
2
votes
0
answers
21
views
Embedding from Transformer-based model from paragraph or documnet (like Doc2Vec)
I have a set of data that contains the different lengths of sequences. On average the sequence length is 600. The dataset is like this:
...
0
votes
1
answer
129
views
Are the Word2Vec encoded embeddings available online? [closed]
I am trying to do an NLP project and was wondering if there is anywhere online where the Word2Vec embeddings are stored (the actual n-dimmensional vectors).
I want to search up a word and see what its ...
- 23
0
votes
1
answer
40
views
Converting age and sex variables to a 64-unit dense layer
I am studying a preprint for my own learning (https://www.medrxiv.org/content/medrxiv/early/2020/04/27/2020.04.23.20067967.full.pdf) and I am befuddled by the following detail of the neural network ...
- 111
3
votes
1
answer
197
views
How does the embeddings work in vision transformer from paper?
I get the part from the paper where the image is split into P say 16x16 (smaller images) patches and then you have to ...
- 233
1
vote
0
answers
24
views
Is there any research work that shows that we should explicitly mark the word boundaries for 1D CNNs?
I'm doing character embedding for NLP tasks using one-dimensional convolutional neural networks (see Chiu and Nichols (2016) for the motivation). I haven't found any empirical evidence of whether or ...
2
votes
1
answer
98
views
In the machine learning literature, what does it mean to say that something is "embedded" in some space?
In the machine learning literature, I often see it said that something is "embedded" in some space. For instance, that something is "embedded" in feature space, or that our data ...
- 423
1
vote
1
answer
163
views
What is the role of embeddings in a deep recurrent Q network?
When describing the model architecture for a deep recurrent q network, the authors of the paper Learning to Communicate with Deep Multi-Agent Reinforcement Learning
each agent consists of a recurrent ...
- 43
23
votes
5
answers
10k
views
What is the difference between latent and embedding spaces?
In general, the word "latent" means "hidden" and "to embed" means "to incorporate". In machine learning, the expressions "hidden (or latent) space" ...
- 33.8k
1
vote
1
answer
2k
views
How is the constraint $\|f(x)\|_{2}=1$ enforced for the embedding $f(x)$ in the FaceNet paper?
In the FaceNet paper, under section 3.2, the authors mention that:
The embedding is represented by $f(x) \in \mathbb{R}^{d}$. It embeds an image $x$ into a $d$-dimensional Euclidean space. ...
- 181
4
votes
2
answers
140
views
What is the intuition behind how word embeddings bring information to a neural network?
How is it that a word embedding layer (say word2vec) brings more insights to the neural network compared to a simple one-hot encoded layer?
I understand how the word embedding carries some semantic ...
- 189