1
$\begingroup$

In natural language processing, we can convert words to vectors (or word embeddings). In this vector space, we can measure the similarity between these word embeddings.

How can we create a vector space where word spelling and pronunciation can be easily compared? For example, "apple" and "ape", "start" and "startle" are very similar, so they should also be similar in this new vector space.

I am eventually looking for a library that can do this out of the box. I would like to avoid implementing this myself.

$\endgroup$
1
$\begingroup$

If you only need the vector space as a way to obtain a similarity measure, you may want to consider a distance measure instead. Similarity and distance are inversely related: identical words have maximum similarity or zero distance, and as the similarity decreases, the distance increases.

For instance, the Wagner-Fischer algorithm computes the edit distance between two strings of characters. This edit distance takes into acccount insertions and deletions, as in your examples, but also substitutions (for example "gray" vs. "grey").

The article linked above includes pseudocode that should translate easily to actual code.

$\endgroup$
  • $\begingroup$ How is the edit distance related to the actual problem? I think you should spend a few words to argue the usefulness of your suggestion with respect to the actual problem. $\endgroup$ – nbro Apr 18 at 13:47
  • $\begingroup$ Not sure whether you mean relationship between edit distance and similarity or wrt vector space. $\endgroup$ – Jaume Oliver Lafont Apr 18 at 14:05
  • $\begingroup$ How does the edit distance relate to e.g. pronunciation? $\endgroup$ – nbro Apr 18 at 14:25
  • $\begingroup$ The edit distance on raw text measures spelling differences. To measure pronunciation differences we need some phonetic transcription of the text. It seems that distinct measures are needed, to deal properly with homophones ( en.wikipedia.org/wiki/Homophone) $\endgroup$ – Jaume Oliver Lafont Apr 19 at 6:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.