I have series of sensors (around 4k) and each sensor will measure the amplitudes at each point.Suppose I train the neural network with sufficent set of 4k values (N * 4k shape). The machine will find a pattern in the series of values.If the values stray away from the pattern (that is anomaly) it can detect the point and will be able to say that anomaly is in the 'X'th sensor.Is this possible.If so what kind of neural network should I use?

  • $\begingroup$ Questions. .. Are the sensors correlated in some way or all completely independent? Are they all required, or can you use a subset? $\endgroup$ – DrMcCleod Mar 7 at 18:20
  • $\begingroup$ Also, do you HAVE to use a neural network? There are easier ways to achieve this. $\endgroup$ – DrMcCleod Mar 8 at 7:43
  • $\begingroup$ Neural Network is not mandatory, any algorithm would do.As of now the sensors seem to be independent. $\endgroup$ – Ram Shanker G Mar 8 at 9:28

Without knowing the kind of data and the process generating it it's hard to give a definite answer. In general, I would attempt a network that has as inputs the actual sensor readings, and outputs the expected readings. You train this network by presenting data with errors added as inputs, and correct readings as outputs. It should learn to guess the correct (non-anomalous) readings from a set of actual readings, and you can find the sensors with anomalies by taking the difference between actual and predicted readings. Depending on the kind of anomalies you expect (misreadings such as fluke zero or max values, or actually anomalous states of the system being measured) your training data should be set up to have samples of such anomalies. If there are temporal correlations (for example, temperature readings that change slowly) using a RNN might be helpful but its dimensioning heavily depends on the nature of your measured system. External factors influencing the system (state of a heating mechanism, time of day/year, etc.) could be added at inputs for better predictions. At the end of the day, trial and error is your friend. Start with a simple network and see how well it behaves. Go for more complex networks when you see where the limitations of the simple solution are.

  • $\begingroup$ Actually the sensors have different idle values at different points.So an anomaly for one may be normal for other and we don't have data for anomalies at each point (at some certain points we have). $\endgroup$ – Ram Shanker G Mar 8 at 10:12
  • $\begingroup$ And can you just elaborate the point.Do we have both anomalies and non-anomalies in training? and what I understood is that we have the labels as 'non' anomalous values' ryt? And if the difference between predicted readings and normal readings are high then as per your explanation is it anomaly or not anomaly? $\endgroup$ – Ram Shanker G Mar 8 at 10:20
  • $\begingroup$ It depends on what sort of output is more useful to you. If you simply want to identify anomalous values, then the network should be trained to output binary per-sensor labels. But if you actually want to "remove" the anomaly from the values then the network should be used for regression. I suspect the network for the former use-case (binary classification with simple anomalous/non-anomalous labels) will be easier to train, but this is just a shot in the dark given that I don't know the kind of data you're dealing with. $\endgroup$ – Ali250 Mar 8 at 10:50

Instead of using a neural network, simply sample as many non-anomalous readings from each sensor as you can. If the distribution of the readings from each sensor is approximately normal (check the skew and kurtosis values for the samples from each sensor) then you can work out mean and standard deviation of the samples and, for any future samples, the value of a particular measurement on a normal probability distribution.

(If your measurements do not have a normal distribution, then you can often apply a transform of some sort to the data make it normal).

So, let's say that you have measured a few thousand typical samples for one of your sensors, confirmed that the distribution is normal and calculated the mean and standard deviation of those samples. You can now calculate where on the normal curve your new sample 'x' would be using:

def gaussian(x, mean, std):
    # check for very small standard deviations
    if(2 * std ** 2 == 0):
        return 0.0

    return (1.0/(math.sqrt(2*math.pi) * std)) * math.exp(-((x-mean)**2)/(2*std**2))

At this point, you can make a decision if the value of the of a particular measurement on that distribution is so low as to be anomalous, and inform the user.

Now, you have probably realised that working out exactly how low a value should be to be considered anomalous might be tricky, and you are correct. The solution is to see if you can get some real anomalous data and use that to set the thresholds. Obviously, this might be rather a job if you have 4000 independent sensors...

If you want to know more about Anomaly Detection then I recommend that you take a look at Andrew Ng's introductory lecture series.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.