1
$\begingroup$

I'm new to AI/ML and I want to research and learn about techniques that could help me to solve this complex task. Any hint would be appreciated.

Let me explain it with an example:

Let's look at two columns PUR.SUPPLY.MTL_REQ_HDR_ID and MTL.PO_REQUISITION_HEADERS_TAB.ID. It is likely they are related (one is the FK and the other one is PK).

enter image description here

As a human I was able to do it by doing the following:

  1. I decoded abbreviations,
  2. I identified context (MTL module),
  3. I identified the subject (requisition header),
  4. I identified ID keyword,
  5. I identified irrelevant information (TAB postfix),
  6. I matched words that are not in the exact order,
  7. I estimated which elements/words do not have to match/occur,

I would like to match millions of columns relatively quickly (seconds). I would like algorithm to learn:

  1. what words are likely context,
  2. what are irrelevant,
  3. what are subjects,
  4. ideally learn some patterns (prefixes, postfixes, name formats, etc),
  5. based on user responses - approve/reject match,
  6. build dictionary of abbreviations,
  7. estimate probability...

I know this is a complex task, but maybe you know a technique, tool, library, article, example ... anything that could be helpful? Any help would be appreciated.

Thanks.

$\endgroup$
4
  • 1
    $\begingroup$ In general Bayesian networks/Naive Bayes do well at text processing. As a library sci-kit has an implementation. $\endgroup$
    – A_Arnold
    Commented Jun 13, 2021 at 4:06
  • $\begingroup$ Thanks @A_Arnold, Naive Bayes seems to be a classification function. I don't think my problem is a classificaiton problem. $\endgroup$
    – Bad Pitt
    Commented Jun 13, 2021 at 18:42
  • $\begingroup$ If you already have some of these matches you can try and scale it to new data. i.e. your pk & fk cols would be x1, x2, match would be y1=1 no match y1=0. Do this for some data, then let it predict some data, update that data if the classification is wrong etc. ALT: you may want to look into ICA which can predict topics. cc.gatech.edu/~isbell/papers/isbell-ica-nips-1999.pdf $\endgroup$
    – A_Arnold
    Commented Jun 13, 2021 at 20:23
  • $\begingroup$ thanks @A_Arnold. We will need to test the bayes and research the ICA. That might be a good start. $\endgroup$
    – Bad Pitt
    Commented Jun 14, 2021 at 21:34

0

You must log in to answer this question.

Browse other questions tagged .