4
$\begingroup$

I have a huge text where a paragraph is written about a project execution. I need to parse that data and make sense out of it. For example the text states, "Est. 200k USD has been committed on the event of sales summit on 8th Sep"

By this statement, i need to make sense of it as approx 200k USD is the revenue, There was a sales summit held on 8th Sep, The amount is already committed.

Likewise many such statements will be there. I need to parse and make a conclusion out of it. Is there anything anyone suggest?

What i have tried so far: Am trying to find a text analytics tool which will group the text into few classifications. Next am trying to write my own logic to tag each paragraph to certain master data. Also trying to use a tool by name orange which is currently helping visualize the data, but not make sense of it.

I think i need to be doing something with Machine learning algorithms. Not sure though.

Improve this question
$\endgroup$
  • $\begingroup$ I would kindly request you to post this same question here Cross Validated.For effective feedback. $\endgroup$ – quintumnia Oct 25 '17 at 18:27
  • $\begingroup$ If your dataset fits a common pattern (ie, the sentences look about the same, with mostly the same types of information to be extracted), you could do this with your standard ML techniques (or even regex if the data is regular). If not, you might try some existing solution like IBM Watson, but your milage may vary depending on how "smart" you expect it to be (its not that smart) $\endgroup$ – k.c. sayz 'k.c sayz' Oct 25 '17 at 23:11
  • $\begingroup$ anything available as open source? $\endgroup$ – Alan Oct 26 '17 at 3:59
  • $\begingroup$ If I understood correctly, check this: github.com/Microsoft/Recognizers-Text and samples here: github.com/Microsoft/Recognizers-Text/tree/master/.NET/Samples $\endgroup$ – Jawad Al Shaikh Oct 28 '17 at 16:09
1
$\begingroup$

Analyzing the existing text with language taggers or with Natural Language Toolkits (NLTK) doesn't make sense. Such software is able to search in the text for words and symbols but it will not understand the meaning. A corpus can not analyzed by itself, text is the output of an underlying model.

At first, the domain has to be converted into a conversational model. The example sentence is from “project execution”, that means an “Enterprise Business Model” is needed. The model can be seen as a game engine which is able to execute natural language. In the literature this idea is called language grounding. It can be realized by handcrafted ontologies or by Recursive neural networks (RNN). The first option (ontologies) are created like a textadventure in a version control system. In the game, the player can commit costs for an event and is able to win or lose. RNN (the second option) for grounded language are called sequence-to-sequence (seq2seq) models and were trained by example data.

The precondition for validating text is a game engine aka language model. That means, a parser can check an input sentence against the model. The bottleneck is, that for most domains such a model is not available. It has to be created from scratch. At first, a model is created from the sample text and then the text can be executed inside the model.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.