0
$\begingroup$

I am working with tabular data that is similar to the below:

Name Phone Number ISO3 Country Amount Email ... ... Outcome Possible Reason
Leona Sunfurry (555)-555-5555 United States 58.96 [email protected] ... ... 0 Not ISO3 country
Diana Moonglory (333)-555-5555 USA 8.32 [email protected] ... ... 1
Fiora Quik (111)-555-5555 FRA 0.35 null ... ... 1
Darius Guy 12345678901234 CAN 555.01 null ... ... 0 Too many digits in phone
LULU (333)-555-5555 CAN 0.00 null ... ... 0 Odd name format
Eve K. (111)-555-5555 FRA 69.25 [email protected] ... ... 1
Lucian Light (999)-555-5555 ENG 65.00 null ... ... 1
Lux D. (333)-555-5555 USA 11.64 [email protected] ... ... 1
Jarvin Crown (333)-555-5555 USA 1357.13 [email protected] ... ... 0 Unknown reason

The table contains information about users. Some of the fields are user-generated while others are generated by the program (like device location, amount, etc.). When this data is collected, it is sent to third parties (we will say a bank). Sometimes the bank rejects the data and it is not good for our users. The rejection could have happened because the user did not input the data correctly or the banks did not like how a field is formatted despite the data being correct and acceptable to other banks.

So we want to find the fields that are causing the most errors and how to fix the issue.

Does it make sense to do pattern recognition on the values to find the reason why the row was rejected? It would need to be an alpha-numeric type of algorithm, it seems.

We know the outcomes from the bank which is labeled as Outcome. Although we have labeled data, it still feels like we need an unsupervised learning algorithm because we do not have labels on why the rows of data were rejected.

Does anyone know what type of algorithm would be best? Any feedback would be appreciated!

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

You should first segregate the rejected samples. You can use then use string matching or something more complex (like creating embeddings and then, taking L2 distance between them) between the different field names you have and the comment for rejection. Whichever field gets the highest score, you increase the rejection count for that field. In the end, you will have a tally of who is your biggest enemy.

You can create some rules which prevent injection of wrong data (like your password should be 7 characters long or something along these lines) or post-process your entries to match a uniform format.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Okay, I believe I understand. Are you familiar with an L2 library or functions in Python by any chance? Also, I tried using correlation coefficients to find which fields correlated most highly with the outcome. Do you have an opinion on that option, albeit less sophisticated than an actual algorithm? $\endgroup$
    – Petar Luketina
    Feb 25, 2021 at 22:00
  • 1
    $\begingroup$ L2 distance you can easily find using numpy's linalg.norm or in scipy.spatial.distance.euclidean. But, you need to convert textual data to numbers. Fields which have negative correlation, if the outcome goes, they down, then they have an antagonistic relationship. Just use string matching and find the bad variables that is good starting approach. Afterwards, you need to devise some rules or methods things, that will take more time. $\endgroup$
    – Abhishek Verma
    Feb 25, 2021 at 22:07

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .