3
$\begingroup$

In the website the following explanation is provided about Embedding layer:

The Embedding layer is initialized with random weights and will learn an embedding for all of the words in the training dataset.

It is a flexible layer that can be used in a variety of ways, such as:

It can be used alone to learn a word embedding that can be saved and used in another model later. It can be used as part of a deep learning model where the embedding is learned along with the model itself. It can be used to load a pre-trained word embedding model, a type of transfer learning.

Isnt embeddings model specific? I mean to learn a representation of something we need the model that something was used to represent! so how can embeddings learned in one model can be used in another?

$\endgroup$

3 Answers 3

2
$\begingroup$

An embedding layer is a linear layer that is used to convert a discrete input into a vector of a fixed size, d. Learned embedding layers are often used in natural language processing (NLP). Common learned embeddings are GloVe, GoogleNews, or word2vec. These embeddings have often been trained on huge amounts of data (eg 3 billion words for GoogleNews), and as such can be applied over a wide variety of contexts (although you might need specific embeddings for specific tasks). These embeddings were all obtained by training neural networks such that words that occur in similar contexts have a similar embedding (you may look up the individual training algorithms for a better understanding).

Words that are similar to each other are closer to each other in d-dimensional space (where d is the vector size), whereas dissimilar words are far apart. This distance can be measured by cosine-similarity. Research has shown that using pre-trained embeddings usually improves model performance in many NLP tasks (https://arxiv.org/pdf/1804.06323.pdf).

If you do use pre-trained word embeddings, make sure that the vocabulary of your training data is present in the embeddings. During neural network training when using pre-trained embeddings, it makes sense to freeze the weights of the embedding to prevent superfluous gradient computation (and ruin your embeddings). However, if some words present in your training data are not present in the embedding, you may consider only partially freezing the weights of your embedding.

$\endgroup$
14
  • $\begingroup$ I think you misunderstood my question, I am asking that how embeddings learned in one model can be used in another? shouldnt embeddings be model specific? $\endgroup$
    – Oculu
    Oct 20, 2022 at 22:44
  • $\begingroup$ I'm afraid I don't understand what you mean. An embedding is simply a linear layer, you can just load its weights. See: keras.io/examples/nlp/pretrained_word_embeddings under the section "Load pre-trained word embeddings" for an example in Keras. $\endgroup$ Oct 20, 2022 at 22:54
  • 1
    $\begingroup$ I have put in the quotation that as per the the website it is mentioned It can be used alone to learn a word embedding that can be saved and used in another model later this is what my concern is how it can be used in another model? Shouldnt the embeddings learned is model-specific i.e. it can be used only in that model? $\endgroup$
    – Oculu
    Oct 20, 2022 at 23:54
  • 1
    $\begingroup$ No, embeddings aren't model-specific. Why would they be? $\endgroup$ Oct 21, 2022 at 0:03
  • 1
    $\begingroup$ I'm on the same page of @Oculu , I don't understand how embeddings can be so model friendly. I get how back propagation works, the point is that any model at the end of the training will generate different weights based on their trainingset and specification, so even if the layer size of different model matches, the weights in that layer doesn't, so how can an embedding (so as you say the first layer of the model) be compatible with the first level of another model? Did we have any reference to a specific explaination about this? $\endgroup$ May 4, 2023 at 12:41
3
$\begingroup$

To answer your one question: Are embeddings model-specific? YES! They are. I am not going to invoke math or other techniques here. My explanation is going to be in a intuitive perspective. I don't know if the current literature and jargons will agree to my usage. But I get your question.

Take a scenario where you trained a CNN to classify smileys or emjoies based on their positivity or negativity. So it's a binary classification problem. Say you achieved a very good model. The penultimate layer for this model will give you a higher dimensional "embedding" vector for any new emojie or smiley pic you are feeding to this CNN.

Can you compare these embedding with this another ANN model's results arising from the same training and testing data? Technically you can't.

But by coincidence it might be very related. How do you verify it? You can use the argument of @postnubilaphoebus to do a PCA on the embedding data so that you can compare these two embeddings (by reducing the dimensions to match between both embeddings results). And then if all inputs which you feed to both models and then PCA-ing the corresponding intermediate penultimate layer outputs gives you pairs of "embeddings". Do a dot product analysis to compare them. You will know if they say the same thing or are very different.

I am not a NLP or LLM person, but this philosophy will work for any models. Thank you. I hope you got an idea. If not I am willing to explain it further.

$\endgroup$
2
  • $\begingroup$ AI noob here. What's a PCA? $\endgroup$
    – darKnight
    Aug 28, 2023 at 18:57
  • $\begingroup$ PCA = Principal Component Analysis. en.wikipedia.org/wiki/…. $\endgroup$
    – dexterdev
    Sep 1, 2023 at 6:19
0
$\begingroup$

Vector Embedding is always model-specific. Model to Model embedding may change.

Please look at this video: https://youtu.be/PR7xz5vQKGg?t=367 and this page which has a diagram showing how the model generates its embedding: https://www.pinecone.io/learn/vector-database/

The idea is to first create an embedding of your content using a specific model and store it in Vector DB.

Then you will use the model again to convert your query (questions you might want to ask to search in your content) to generate embedding using the same model and use that embedding query matrix to search vector DB.

The model controls how to generate embedding. It depends on what that model is supposed to do. It will create embedding accordingly on content as well as query.

Remember this "Embeddings differ from other machine learning techniques in that they are learned through training a model on a large dataset rather than being explicitly defined by a human expert. This allows the model to learn complex patterns and relationships in the data that may be difficult or impossible for a human to identify.". In short, each model may generate different embeddings. We don't control how the model will generate its embeddings. The model evolves as it is trained on a dataset. Once the Model is finalized, it is settled on a specific type of embedding. https://encord.com/blog/embeddings-machine-learning/

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .