What machine learning algorithm should be used to analyze the relationship between strings?

Question

I am trying to build a neural network that takes in a single string, ex: "dog" as an input, and outputs 50 or so related hashtags such as, "#pug, #dogsarelife, #realbff".

I have thought of using a classifier, but because there is going to be millions of hashtags to choose the optimal one from, and millions of possible words from the english dictionary, it is virtually impossible to search up the probability of each

It is going to be learning information from analyzing twitter posts' text, and its hashtags, and find which hashtags goes with what specific words.

Answer 1

Here is a good approach to achieve the task you want:

Step 1- Compute the Vector representation (i.e embeddings) of all the words you want to include. There are many algorithms out there to achieve this task.

Step 2- Choose the #words corresponding to your input word (e.g dog) by applying K-Nearest Neighbors (KNN) or similar algorithms. You basically compute the distances using the embeddings.

Steps Detailed:

Step 1-

In NLP we represent human language as a vector of values instead of a set of characters in order to process it. To do so there are 3 approaches in the literature:

- Word Level Embeddings: Represent each word as a vector of values. Algorithms: Word2Vec by Google (paper), fastText by Facebook, GloVe by Stanford University (paper) ...

- Character Level Embeddings: Represent each character as a vector of values. Algorithms: ELMo (paper) ...

- Sentence Level Embeddings: Represent a sentence as a vector of values. Algorithms: Universal Sentence Encoder by Google (paper) ...

In your case I suggest to use GloVe or ElMo if you have only words and Universal Sentence Encoder if you have words and sentences. . Compute all your words embeddings and move to the next step.

Step 2-

Now that you have your embeddings, compute the distances between all your words (use Euclidian, Minkowski or any other distance). Notice that the computation may take some time but will only be executed once.

Now each time you have a word (e.g dog) you apply the KNN algorithm using the computed distances and you will get the most related words to this word.

Note: No need to compute distances and apply KNN if you use Universal Sentence Encoder as the similarity is easily computed using a dot product of the embeddings. See my quick implementation example here for details.

Answer 2

You can try using Mallet (which uses Gibs Sampling) or gensim LDA (which uses drichlet priors) to model the problem as topics (hashtags) in different documents (tweets). https://towardsdatascience.com/topic-modeling-and-latent-dirichlet-allocation-in-python-9bf156893c24 has a nice example.

Answer 3

I think your intuition about a classifier being the wrong approach is a good one. This looks like a great use-case for word vectors, a "self-supervised" learning technique that maps tokens (e.g. "dog") to vectors (which might have anywhere between 50 - 500 dimensions). Facebook open-sourced a particular excellent tool for training word vectors called FastText; you could use this to embed tokens and hashtags alike into a word embedding space. You should find that the vector for "dog" ends "close to" (small cosine distance). Given a word, you can easily look up its vector (after training on your corpus, of course), but how to find other vectors that are close to it? If you want to do better than "brute force" and you need to check against a large number of (vectors for) hashtags, you should consider using Facebook's excellent FAISS library for fast similarity search to find the closest hashtags.