1
$\begingroup$

I have a bunch of text documents, split into source documents and transformed documents. These text documents have multiple lines and are edited at specific locations, in a specific way.

I make use of the difflib package available in Python to identify the associated transformation, for each source document and the resulting transformed document.

I wish to train and implement a ML technique which will help in identifying and automating this conversion activity.


Here is a sample result of how the transformation result looks like: (NOTE: This example contains only one line, but my actual use case contains several lines)

import difflib

Initial = 'This is my initial state'
Final = 'This is what I transform into'

diff = difflib.SequenceMatcher(None, Initial, Final)

for tag,i1,i2,j1,j2 in diff.get_opcodes():
    print('{:7} Initial[{:}:{:}] --> Final[{:}:{:}] {:} --> {:}'.format(tag,i1,i2,j1,j2,Initial[i1:i2],Final[j1:j2]))

#Result:
equal   Initial[0:8] --> Final[0:8] This is  --> This is 
insert  Initial[8:8] --> Final[8:23]  --> what I transfor
equal   Initial[8:9] --> Final[23:24] m --> m
delete  Initial[9:10] --> Final[24:24] y --> 
equal   Initial[10:13] --> Final[24:27]  in -->  in
delete  Initial[13:14] --> Final[27:27] i --> 
equal   Initial[14:15] --> Final[27:28] t --> t
replace Initial[15:24] --> Final[28:29] ial state --> o

This helps in outlining the transformation steps to transform Initial to Final. I wish to make use of ML to identify the common pattern in such transformation between a large collection of txt documents and train a model that I can use in future.


What will be the best method to approach this problem? I am not facing a problem in identifying and classifying text data, but in identifying the nature of editing and transformation of strings.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.