I have a bunch of text documents, split into source documents and transformed documents. These text documents have multiple lines and are edited at specific locations, in a specific way.

I make use of the difflib package available in Python to identify the associated transformation, for each source document and the resulting transformed document.

I wish to train and implement a ML technique which will help in identifying and automating this conversion activity.

Here is a sample result of how the transformation result looks like: (NOTE: This example contains only one line, but my actual use case contains several lines)

import difflib

Initial = 'This is my initial state'
Final = 'This is what I transform into'

diff = difflib.SequenceMatcher(None, Initial, Final)

for tag,i1,i2,j1,j2 in diff.get_opcodes():
    print('{:7} Initial[{:}:{:}] --> Final[{:}:{:}] {:} --> {:}'.format(tag,i1,i2,j1,j2,Initial[i1:i2],Final[j1:j2]))

equal   Initial[0:8] --> Final[0:8] This is  --> This is 
insert  Initial[8:8] --> Final[8:23]  --> what I transfor
equal   Initial[8:9] --> Final[23:24] m --> m
delete  Initial[9:10] --> Final[24:24] y --> 
equal   Initial[10:13] --> Final[24:27]  in -->  in
delete  Initial[13:14] --> Final[27:27] i --> 
equal   Initial[14:15] --> Final[27:28] t --> t
replace Initial[15:24] --> Final[28:29] ial state --> o

This helps in outlining the transformation steps to transform Initial to Final. I wish to make use of ML to identify the common pattern in such transformation between a large collection of txt documents and train a model that I can use in future.

What will be the best method to approach this problem? I am not facing a problem in identifying and classifying text data, but in identifying the nature of editing and transformation of strings.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.