2
$\begingroup$

I have data that is collected from several different instruments simultaneously that is generally analyzed on a location-by-location basis. A skilled interpreter can identify "markers" in the data that represent a certain change in conditions with depth - each marker only occurs once in each series of data. However, it is possible that a maker is absent either due to missing data or that physical condition not existing.

Often, there are dozens of these markers per location and thousands, if not 10's of thousands of measurements that need to be interpreted. The task is not that difficult and there are many strong priors that can be used to guide interpretation. E.g., if marker A is in location #1, and location #2 is very close to location #1, it is likely that marker A will be present in a very similar relative position. Also, if you have markers A, B, and C they will always be in that order. Although it could be that you have A/B/C or A/C or B/C, etc.

I am including a hand-sketched example below with 4 example locations and one data stream (I normally have 4-5 data streams per location.

I am looking for guidance on the type of algorithm to apply to this problem. I have explored Dynamic Time warping, but the issue is that with 10-20k data samples per location, and thousands of locations, the problem becomes computationally challenging.

Also, in general you may have 10000 locations, with maybe 100 that have been hand interpreted by an expert.

hand sketch of example data

$\endgroup$
  • $\begingroup$ Where do the markers come from? Are they in the data, or do you need to spot them? I can't quite make sense of what it is you're trying to achieve. $\endgroup$ – Oliver Mason Apr 26 at 17:00
0
$\begingroup$

Thresholding based on variance is one method for anomaly detection in time-series data.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.