What is the best way to create a vector representation (with fasttext) of a list of words?

Basically what I want to do is to create a single vector representation of a list of skills belonging to employees at a company (one list per employee). The embedding will be a representation of an employee's "profile". The motivation behind this is (among other reasons) that I want to be able to identify clusters among the employees.

Assume I already have a trained FastText model (or Word2vec) that can generate good representations of the individual words in the list.

My current solution is simply to add all the word embeddings in an employee's list together (without any form of normalization). But I'm very unsure about whether this is the best approach to generating a good representation of an employee's profile.

The dimensions of the vectors are 300 and there are usually around 10 to 30 skills in a single list.

Any help would be greatly appreciated!

Example:

Let's say we have an it-consulting firm where each employee has their own set of skills. Some consultants are more experienced or versatile, thus having more skills listed in their profiles. eg we have:

 alex_skills = ['microsoft azure', 'machine learning', 'data science', 'python', 'sklearn', 'xgboost', 'nginx', 'flask', 'SHAP', 'git', 'word2vec', 'statistics', 'deep learning', 'linux','docker compose', 'pandas']

carla_skills = ['devops', 'machine learning', 'deep learning', 'continuous integration', 'kubernetes', 'python','git', 'speech recognition', 'github', 'bitbucket', 'scikit-learn', 'natural language processing', 'pandas']

adam_skills = ['automation', 'robotic process automation', 'banking and finance', 'process mapping', 'IAM', 'väsentlighetsanalys', 'business intelligence', 'auditor', 'requirements handling', 'risk management', 'coordinator', 'project manager', 'data visualization']


As you can see Alex and Carla are more similar and should possibly be in the same cluster, while Adam might not be.

So I wan't to make a vector representation of the entire list of skills. And then I will use these vector representations in some clustering algorithm (eg HDBscan) and by some distance metric (eg. cosine distance), capture the relation between Alex and Carla.

I suspect the fact that the lists have different lengths might cause problems, therefore maybe divide by the length of the list after adding?

• Hi @Love Book and welcome to AI Stack Exchange! For clarification, could you define your notion of 'best' and 'good' in this question? For example, are you wondering about a specific use case after the representation/embedding is created, or are you wondering about the most standard/general way of creating the representation (or something else)? We hope to see more of you on this site! Jun 13 at 15:59
• Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer.
– Community Bot
Jun 13 at 15:59

Something that might work is a Universal Sentence Encoder. It's not quite what you asked for, as it generates encodings directly from text, not from other word vectors, but it is designed for matching related text:

The researchers who developed it claim the generated "sentence embeddings can be trivially used to compute sentence level semantic similarity scores that achieve excellent performance on the semantic textual similarity (STS) Benchmark". (It may not work as well on your task, of course.)

It's available on the Tensorflow Hub, with a demo on Colab, or you can read the research paper here.

Welcome to the community.

So basically you would like to come up with a single vector representation for each employer considering different skills that particular employer has?

If that is the case, I would say it depends on what would be your goal after doing this analysis. I have seen papers where they simply take the average of these different vectors or just sum them up. Try different approaches and see if it works in your downstream analysis.

You mentioned that one motivation is to identify clusters among employers. I suggest you try averaging and summing them up and analyzing your downstream task (clustering) to see how each approach works.

I will give you an example: In the language of life domain (DNA, RNA, amino acid sequences), there is a paper which describes SeqVec, a tool that utilizes ELMo language model to train and provide vector representations for protein sequences.

Say you have a protein sequence as follows: HMGHGHHMGKGAAKKHMALLKGGAGLCMLK

Now tokenize it in 3-mers: HMG MGH GHG HGH ...

Then imagine we get a vector representation of real numbers for each of these 3-mers.

Then if you want to come up with a representation for the whole protein sequence, would you get the sum of all 3-mers or get the average of it or do something else? I would say, it depends on what problem you are solving and trying different approaches to see which one works in your case.

Hope my answer helped you a little bit. Cheers.