4
$\begingroup$

I have a dataset with missing values, I would like to use machine learning methods to fill. In more detail, there are $n$ individuals, for which up to 10 properties are provided, all numerical. The fact is, there are no individuals for which all properties are given. The first rows (each row contains data for a given individual) do look as the following

\begin{bmatrix} 1 & NA & 3.6 & 12.1 & NA \\ 1.2 & NA & NA & 4 & NA \\ NA & 4 & 5 & NA & 7 \end{bmatrix}

What methods could be applicable in general?

I have some basic experience in classifiers and Random Forests. Modulo the obvious difference that this is not a classifying problem, what I struggle most with is that the same variable (described in the e.g $n$-th column) is both an input and an output. Say I want to predict the value $A_{2,3}$ in the dataset above. In this case, all the values in the third column could be used as input, excluded of course $A_{2,3}$ itself, which would be an output.

This seems to be different than the more conventional set-up of predicting a property, given a set of other properties (e.g, predict income given education, work sector, seniority, etc.). In this case, sometimes the income is to be predicted, sometimes used for predicting another variable. I am aware of methods which, given a vector $X_i$, could approximate a function $F$ and predict responses $Y_i$ with

$$ Y_i = F(X_i)$$

In the scenario I described though, it looks like some implicit function $\Phi$ is to be found, a function of all the variables $Z_i$ (columns in the dataset above)

$$ \Phi (Z_i) = 0$$

What methods could handle this aspect? I understand the question is probably too general, but I could not find much and could do with a starting point. I would be already content with some hints for my further reading, but anything more would be gratefully welcomed, thanks.

$\endgroup$
  • $\begingroup$ Hi and welcome to AI SE! Have you tried the typical methods of filling the missing values with the average (of the non-missing values in the corresponding column/property), zero, minimum, and maximum? I don't know if this will work, but that should be a starting point. $\endgroup$ – nbro May 15 at 10:24
  • $\begingroup$ Yes that is what I have done so far, but it seems really inaccurate, as it grossly misses some obvious correlations. For example, if an individual is very tall, their weight will also tend to be larger. But if my dataset is populated mainly by short individuals, the average will be skewed and tall individuals will be assigned a smaller weight. $\endgroup$ – Smerdjakov May 15 at 10:43
  • $\begingroup$ Hi, could you explain the data set more, what it represents? $\endgroup$ – Igor May 17 at 11:11
  • $\begingroup$ @Igor, frankly the question is very general, I do not have a specific dataset in mind, more a general problem, which I thought must be quite common in Data Science. I re-read my post and the beginning might be confusing indeed.. $\endgroup$ – Smerdjakov May 18 at 18:27
1
+50
$\begingroup$

As you mentioned in the comments about a possible problem of using mean, median type of imputations naively could lead to wrong predictions. In such cases, you need to first check whether you have enough data.

If you have enough data

You can try using MICE (Multivariate Imputation By Chained Equations) algorithm on your missing data. The method is based on Fully Conditional Specification, where each incomplete variable is imputed by a separate model. The MICE algorithm can impute mixes of continuous, binary, unordered categorical and ordered categorical data. One note of caution with this method: It is a computationally expensive method, so it use it if you are not short on time.

The important thing to keep in mind is that in order to tackle such problems, you would be needing multiple iterations as a part of your algorithm. The conventional setup you are describing, does not seem to iterative in nature and hence you are running into the problem of features being input and output at the same time.

If by any chance you insist on finding the missing values just for solving a downstream task like classification or regression, you can try the XgBoost algorithm. It can be used as a classifier or as a regressor. This algorithm can handle missing values inherently. Source: this answer

If you don't have enough data

In such a case you would need to introduce bias in your model using your insights or domain knowledge about the problem. For, e.g., in the a possible problem of estimating weights using heights, you had an insight that your data comprises more of short people. So instead of naively using median values of the total dataset for weight, you can try to bin the data according to their heights, say 'S', 'M', 'L', 'XL', and estimate the weights of each bin separately using the median values of their respective bins. The thing to keep in mind is that when data is low, you need to provide knowledge to the model by enforcing some bias using your insights and domain knowledge about the problem.

| improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ Thanks, very useful references. $\endgroup$ – Smerdjakov May 23 at 11:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.