How do you build a language model to predict the contextual similarity between two documents?

Question

How do you build a language model to predict the contextual similarity between two documents?

Answer 1

You can use an autoencoder for text. For example, you can refer to this example here: https://machinelearningmastery.com/lstm-autoencoders/

For comparing the contextual similarity, you can compare the encoded vectors for the distance maybe through a formula like mean squared error.

This works as the autoencoder compresses the input data into a vector of numbers, forcing the encoder and the decoder to learn specific features about text. The features cannot be understood by humans but have meanings.

Another approach will be using some word or document embedding like word2vec or GloVe. IT may also work well depending on your dataset size. You need to experiment through different methods to find out which is the best.

A supervised method can also be used if you have labels. For example, you can use a LSTM and train it like an Siamese network for LSTM using triplet loss. Here is an example: https://medium.com/@gautam.karmakar/manhattan-lstm-model-for-text-similarity-2351f80d72f1 Implementation: https://github.com/GKarmakar/deep-siamese-text-similarity

Hope this can help you and have a nice day