I am developing an AI tool for anomaly detection in a distributed system.  The system supports an interface that combines several individual logs into a single log file generating approx. 7000 entries/min. The logs entries are partially system generated (d-Bus, IPC, ….)  and human written statements (Status not received, initialized successfully, ….). The developers use the generated log for debugging. The entries have been configured to have a similar format depending on the generated system (timestamp, ids, component, context, verbosity level, description, ….). 

1. The history of the identified anomalies is minimal and not archived.
2. Not many similar event templates in log files.
3. Software execution rules are not clearly documented.
4. The log events are co-related.

What are the recommended algorithms (Statistical, NLP, ML, Neural networks) that can be used to efficiently perform pattern extraction on the entries and identify existing and new anomalous behavior?

  • $\begingroup$ Are your categories only "anomalous" and "not anomalous" or you want to perform anomaly detection and categorisation (as two different tasks)? $\endgroup$
    – nbro
    Feb 24 '19 at 16:28
  • $\begingroup$ The main goal is to perform anomaly detection. By categorization, I meant the extraction of features from the log events relevant to the identification of the anomalous behavior. $\endgroup$
    – Ben
    Feb 24 '19 at 17:28
  • $\begingroup$ Is the data going to be continuously provided (that is, will you keep receiving a stream of data) or is the data contained in a set which will not change? $\endgroup$
    – nbro
    Feb 24 '19 at 17:29
  • $\begingroup$ I am looking into the stream of data already present(offline). But in the future, it is desired to extend the method to perform on a stream of incoming data (online). $\endgroup$
    – Ben
    Feb 24 '19 at 17:34

In the paper "Unsupervised real-time anomaly detection for streaming data" (by Subutai Ahmad, Alexander Lavin, Scott Purdy and Zuha Agha), 2017, an algorithm for anomaly detection (particularly suited for cases where a stream of data is continuously provided) is described. This algorithm is based on Numenta's Hierarchical Temporal Memory model.

I've actually never used it, but I know that Numenta's work is particularly suited for anomaly detection. You can have a look at it and see if it fits your needs. Have also a look at the Numenta Anomaly Benchmark (NAB).

  • $\begingroup$ Thank you for the reference. I am new to machine learning related topics. I am a bit confused here about the datasets(logs) I have. Do I need to start by thinking about how to label them (anomalous/normal behavior)? Or can I perform anomaly detection on the unlabeled dataset? $\endgroup$
    – Ben
    Feb 24 '19 at 20:01
  • $\begingroup$ @Ben In this case, I don't think you will need labelled data. Anomaly detection only consists in finding certain patterns in the data (so this is a unsupervised learning technique). $\endgroup$
    – nbro
    Feb 24 '19 at 20:19
  • $\begingroup$ Do you have suggestions for any other unsupervised learning techniques? The Numenta approach cannot be utilized for my work due to license issues. $\endgroup$
    – Ben
    Mar 2 '19 at 12:37
  • $\begingroup$ @Ben I would not have other suggestions (right now). But why is the license an issue? I think they have an open-source option. $\endgroup$
    – nbro
    Mar 2 '19 at 12:38
  • $\begingroup$ Unfortunately, it is blacklisted for use in my company. $\endgroup$
    – Ben
    Mar 2 '19 at 12:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.