0
$\begingroup$

I'm working on data cleaning and I'm stuck. I have a data set with 3 columns: id, age, and weight.

Supposing I have an entry:

id:1 | age:3 (years)  | weight: 150 (kg)

How can we detect that the information is wrong, assuming I have a thousand lines?

And how can I correct it (using Python)?

Is there any function in Python that I can use or should I use machine learning techniques?

$\endgroup$
2
$\begingroup$

While I can see that there are some heuristics that can tell you whether an entry is 'weird', I don't see any way that you can correct this. Where would you get a correct value from?

I would perhaps start with a statistical analysis, looking at the distribution of values to get an idea of the state the data is in. From this you can then already see some values that wouldn't make sense (this depends very much on what data it is: census data will include toddlers, but credit card applications wouldn't).

Your best bet is to encode some constraints, such as a minimum/maximum range for each value, and perhaps some correspondences, such as a maximum weight for a certain age range. This you can do with simple conditional statements in Python.

You can then flag up those entries which don't look quite right. What you do with them depends on your application. You can filter them out, or manually change the values to something that is consistent with possible values.

Machine Learning techniques would not be able to do anything better than that.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thank you for the answer. $\endgroup$ – Asmaa Feb 10 at 21:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.