0
$\begingroup$

I have 2 tabular datasets, one is clean and one is drifted. They are records of sensor measurements. I move the sensor around in the room and collected thousands of measurements.

I have a sensor that is supposed to track a signal from just one main source. But there are many sources that interfere with the main source in the dirty room.

In clean data, I have only one source when recording the measurements.

In dirty data, I have many interference sources when recording the measurements.

E.g. if the clean data has only 2 features, one of the row is 5,10. When it's affected by other sources, their value can be 7 and 8 or something like that. But it's not just a white noise that disappears later. It's a permanent drift that will never be gone unless I eliminate the interfering sources from the room.

That means if I measure the value it will always report 7 or 8 in the same dirty room every single time.

I want to separate out only the main source's measurement. So given 7 and 8, I want the output to be 5,10. But I don't have the input/output pair to train a machine learning model. So this should be an unsupervised learning problem.

My idea is to train an unsupervised model to know what a clean data looks like. Then when given dirty data, it can convert the dirty data to a corresponding clean data like the example I gave. So the model essentially learns to ignore all the other sources and report me the main source only. The number of sources is not known (maybe the clean area I thought was clean is actually containing other sources but I am fine if the model can convert the dirty to the almost clean data I provided)

Please give me a link or topics about this or explain your idea on how to make this unsupervised model.

PS. There are a lot more than 2 features for the sensor, that's why I think it's possible to know the clean data from the dirty one. And I also have time series data as I move the sensor around collecting measurements, let me know if you know how to make use of that to clean the data.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.