2
$\begingroup$

I have thousands groups of paragraphs and I need to classify these paragraphs. The problem is that I need to classify each paragraph based on other paragraphs in the group! For example, a paragraph individually maybe belongs to class A but according to other paragraph in the group it belongs to class B.

I have tested lots of traditional and deep approaches( in fields like text classification, IR, text understanding, sentiment classification and so on) but those couldn't classify correctly.

I was wondering if anybody has worked in this area and could give me some suggestion. Any suggestions are appreciated. Thank you.

Update 1:

Actually we are looking for manual sentences/paragraph for some fields, so we first need to recognize if a sentence/paragraph is a manual or not second we need to classify it to it's fields and we can recognize its field only based on previous or next sentences/paragraphs.

To classify the paragraphs to manual/no-manual we have developed some promising approaches but the problem come up when we should recognize the field according to previous or next sentences/paragraphs, but which one?? we don't know the answer would be in any other sentences!!.

Update 2:

We can not use whole text of group as input because those are too big (sometimes tens of thousands of words) and contain some other classes and machine can't learn properly which lead to the drop the accuracy sharply.

Here is a picture that maybe help to better understanding the problem: enter image description here

|improve this question
$\endgroup$
  • $\begingroup$ You clearly need to classify using data from the whole group. What have you tried? Are you only using data from the single paragraph so far? If so, what is preventing you using data from the whole group of paragraphs? $\endgroup$ – Neil Slater Jul 29 at 14:22
  • $\begingroup$ The whole group of paragraphs are too big! sometimes thousands of words and with traditional text-classification and even deep learning I used before the accuracy dropped sharply when I'm using the whole group of paragraphs $\endgroup$ – Sina Jul 29 at 14:26
  • $\begingroup$ But you agree you must use some data from the surrounding paragraphs? Because you are only able to identify the correct classification by looking at them? Your ML needs that same data somehow to perform the task you want, at least according to your description. Please add the details from your comment to the question - use edit and explain why you cannot simply train on all data. $\endgroup$ – Neil Slater Jul 29 at 14:45
  • $\begingroup$ Sure, Thank you for suggestion! $\endgroup$ – Sina Jul 29 at 14:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.