1
$\begingroup$

I have a corpus, say an instruction manual. The text in this manual is grouped into chapters and each chapter is split up into sections. For example, Chapter 1/Section 1, Chapter 1/Section 2 and so on. Assume the corpus has C chapters and each chapter has S sections. My goal is, given a sentence or question, to classify this sentence/question. In other words I want to compute three most probable chapters to which this sentence or question belongs to. I tried MultinomialNB model using sklealrn, but it did not give me the desired result. I want to try another approach, for example using a Neural Network and compare it with the MultinomialNB model. I have Googled and found Doc2Vec but haven't tried yet.

Can anyone suggest a better or another possible approach so that I could try and compare? What is the standard approach to such kind of problem?

Improve this question
$\endgroup$
0

2 Answers 2

Reset to default
1
$\begingroup$

You have got text which have words. Now each of the words have dependency with words that are occurring previously or/and with the ones occurring later. So to capture the "context" in which a particular word occurs is done by sequence models in deep learning. Sequence models that are quite used often are Recurrent Neural Networks (RNN), Gated Recurrent Units (GRU) and Long Short Term Memory (LSTM) listed in the order of their functionality and complexity. To know more about them visit Jonathan Hui's blog.

Your problem is a classification problem i.e. given a sentence the classifier should tell which chapter and section it belongs to. You can use one of the sequence models to "encode" the sentence as a vector (based on the context of words in it and meaning of sentence) and pass that vector to a fully-connected neural network that has a softmax layer at the end to tell which chapter and section the text belongs to. The number of classes you are having are C * S (one for each chapter-section pair). From the output of softmax layer you can pick up the top 3 classes that got the highest probability for a sentence.

Doc2Vec converts a paragraph (can also be a sentence) into a vector representation. The Doc2Vec model has been trained over a huge corpus of text to be able to capture the context of a text and represent it as a high dimensional vector.

The corpus of text that you have, is from a probability distribution is quite different from that used for training Doc2Vec model. Sequence models (mentioned above) can learn that probability distribution for your text corpus exclusively. In other words, you have a tailor-made representation for your text so that the context can be understood much better.

That is why I suggested you to use sequence models. You can also compare the performance obtained by using pretrained Doc2Vec as text representation with that obtained by using one of the sequence models.

To know about the effectiveness of RNN visit Andrej Karpathy's blog.

Improve this answer
$\endgroup$
3
  • $\begingroup$ I have another question. As far as I know there are pretrained models for Word2Vec, e.g., Google's Word2Vec or Standford Univ's GloVe. We can download those models and use it. However, libraries such as Keras/Gensim provide with means for Word2Vec using which we can train our own model by feeding with labeled sentences from our corpus. Also, Word2Vec (shallow) neural network is well known and proved to be efficient (in the scientific circles). The same for Doc2Vec. $\endgroup$
    – HardFork
    Jun 21, 2018 at 1:49
  • $\begingroup$ So, wouldn't it be better to exploit the underlying Doc2Vec neural network to train my model using my domain specific corpus? Or should I try different NN architectures of RNN, LSTM and etc and later compare them? I mean should I spend my time on what people have already proved? $\endgroup$
    – HardFork
    Jun 21, 2018 at 1:49
  • $\begingroup$ Its worth doing both so that you can compare the extent to which neural nets in Doc2Vec and RNNs can model dependencies between words. $\endgroup$
    – varsh
    Jun 21, 2018 at 5:22
1
$\begingroup$

The easiest way would probably be not to use machine learning at all. Create an inverted index of words (ie for each word occurrence record the chapter and section), and then use an information retrieval algorithm (eg TF-IDF) to find the section that best matches your question.

This will be more efficient (no huge models to train) and more transparent (you can easily see why a particular section has been selected).

Additional steps you can take to adjust performance is stemming/lemmatising of words, and adding synonym lists (eg in a car manual application you might want to treat trunk and boot as the same word).

I have used that approach recently in a commercial project, and it performed well.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thank you very much for your suggestion. I will try it. Unfortunately cannot upvote since I do not have enough reputation. $\endgroup$
    – HardFork
    Jun 20, 2018 at 17:08
  • $\begingroup$ I create tf-idf scores for each word for each document. Sklearn's TfidfVectorizer allows to do it in a couple of lines. But it creates scores for words, not for sentences. As I described, my task is classify a particular sentence. So, I need to come up with a method that computes similar score for sentence. For example given a sentence S, I want to compute C scores, one score for each class (chapter in my case). Do you know any approach how to go about it? Any reference? A naive approach is to sum up scores of each word or compute average over all words in the sentence. $\endgroup$
    – HardFork
    Jun 22, 2018 at 20:52
  • $\begingroup$ @B.K. Yes, I'd sum scores or average, depending on what works better in that particular application. Adding favours longer sequences, so if that is a problem you could average instead. $\endgroup$
    – Oliver Mason
    Jun 25, 2018 at 8:18

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .