I have 2 tabular datasets, one is clean and one is drifted. They are records of sensor measurements. I move the sensor around in the room and collected thousands of measurements.

I have a sensor that is supposed to track a signal from just one main source. But there are many sources that interfere with the main source in the dirty room.

In clean data, I have only one source when recording the measurements.

In dirty data, I have many interference sources when recording the measurements.

E.g. if the clean data has only 2 features, one of the row is 5,10. When it's affected by other sources, their value can be 7 and 8 or something like that. But it's not just a white noise that disappears later. It's a permanent drift that will never be gone unless I eliminate the interfering sources from the room.

That means if I measure the value it will always report 7 or 8 in the same dirty room every single time.

I want to separate out only the main source's measurement. So given 7 and 8, I want the output to be 5,10. But I don't have the input/output pair to train a machine learning model. So this should be an unsupervised learning problem.

My idea is to train an unsupervised model to know what a clean data looks like. Then when given dirty data, it can convert the dirty data to a corresponding clean data like the example I gave. So the model essentially learns to ignore all the other sources and report me the main source only. The number of sources is not known (maybe the clean area I thought was clean is actually containing other sources but I am fine if the model can convert the dirty to the almost clean data I provided)

Please give me a link or topics about this or explain your idea on how to make this unsupervised model.

PS. There are a lot more than 2 features for the sensor, that's why I think it's possible to know the clean data from the dirty one. And I also have time series data as I move the sensor around collecting measurements, let me know if you know how to make use of that to clean the data.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.