1
$\begingroup$

In natural language processing, we can convert words to vectors (or word embeddings). In this vector space, we can measure the similarity between these word embeddings.

How can we create a vector space where word spelling and pronunciation can be easily compared? For example, "apple" and "ape", "start" and "startle" are very similar, so they should also be similar in this new vector space.

I am eventually looking for a library that can do this out of the box. I would like to avoid implementing this myself.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

If you only need the vector space as a way to obtain a similarity measure, you may want to consider a distance measure instead. Similarity and distance are inversely related: identical words have maximum similarity or zero distance, and as the similarity decreases, the distance increases.

For instance, the Wagner-Fischer algorithm computes the edit distance between two strings of characters. This edit distance takes into acccount insertions and deletions, as in your examples, but also substitutions (for example "gray" vs. "grey").

The article linked above includes pseudocode that should translate easily to actual code.

Improve this answer
$\endgroup$
4
  • $\begingroup$ How is the edit distance related to the actual problem? I think you should spend a few words to argue the usefulness of your suggestion with respect to the actual problem. $\endgroup$
    – nbro
    Commented Apr 18, 2019 at 13:47
  • $\begingroup$ Not sure whether you mean relationship between edit distance and similarity or wrt vector space. $\endgroup$
    – Jaume Oliver Lafont
    Commented Apr 18, 2019 at 14:05
  • $\begingroup$ How does the edit distance relate to e.g. pronunciation? $\endgroup$
    – nbro
    Commented Apr 18, 2019 at 14:25
  • $\begingroup$ The edit distance on raw text measures spelling differences. To measure pronunciation differences we need some phonetic transcription of the text. It seems that distinct measures are needed, to deal properly with homophones ( en.wikipedia.org/wiki/Homophone) $\endgroup$
    – Jaume Oliver Lafont
    Commented Apr 19, 2019 at 6:07

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .