# Finding anomaly detection by pattern matching in a set of continous data

I have series of sensors (around 4k) and each sensor will measure the amplitudes at each point.Suppose I train the neural network with sufficent set of 4k values (N * 4k shape). The machine will find a pattern in the series of values.If the values stray away from the pattern (that is anomaly) it can detect the point and will be able to say that anomaly is in the 'X'th sensor.Is this possible.If so what kind of neural network should I use?

• Questions. .. Are the sensors correlated in some way or all completely independent? Are they all required, or can you use a subset? – DrMcCleod Mar 7 '19 at 18:20
• Also, do you HAVE to use a neural network? There are easier ways to achieve this. – DrMcCleod Mar 8 '19 at 7:43
• Neural Network is not mandatory, any algorithm would do.As of now the sensors seem to be independent. – Ram Shanker G Mar 8 '19 at 9:28

• Actually the sensors have different idle values at different points.So an anomaly for one may be normal for other and we don't have data for anomalies at each point (at some certain points we have). – Ram Shanker G Mar 8 '19 at 10:12
• And can you just elaborate the point.Do we have both anomalies and non-anomalies in training? and what I understood is that we have the labels as 'non' anomalous values' ryt? And if the difference between predicted readings and normal readings are high then as per your explanation is it anomaly or not anomaly? – Ram Shanker G Mar 8 '19 at 10:20
• It depends on what sort of output is more useful to you. If you simply want to identify anomalous values, then the network should be trained to output binary per-sensor labels. But if you actually want to "remove" the anomaly from the values then the network should be used for regression. I suspect the network for the former use-case (binary classification with simple anomalous/non-anomalous labels) will be easier to train, but this is just a shot in the dark given that I don't know the kind of data you're dealing with. – Ali250 Mar 8 '19 at 10:50

Instead of using a neural network, simply sample as many non-anomalous readings from each sensor as you can. If the distribution of the readings from each sensor is approximately normal (check the skew and kurtosis values for the samples from each sensor) then you can work out mean and standard deviation of the samples and, for any future samples, the value of a particular measurement on a normal probability distribution.

(If your measurements do not have a normal distribution, then you can often apply a transform of some sort to the data make it normal).

So, let's say that you have measured a few thousand typical samples for one of your sensors, confirmed that the distribution is normal and calculated the mean and standard deviation of those samples. You can now calculate where on the normal curve your new sample 'x' would be using:

def gaussian(x, mean, std):
# check for very small standard deviations
if(2 * std ** 2 == 0):
return 0.0

return (1.0/(math.sqrt(2*math.pi) * std)) * math.exp(-((x-mean)**2)/(2*std**2))


At this point, you can make a decision if the value of the of a particular measurement on that distribution is so low as to be anomalous, and inform the user.

Now, you have probably realised that working out exactly how low a value should be to be considered anomalous might be tricky, and you are correct. The solution is to see if you can get some real anomalous data and use that to set the thresholds. Obviously, this might be rather a job if you have 4000 independent sensors...

If you want to know more about Anomaly Detection then I recommend that you take a look at Andrew Ng's introductory lecture series.