I am developing an AI tool for anomaly detection in a distributed system. The system supports an interface that combines several individual logs into a single log file generating approx. 7000 entries/min. The logs entries are partially system generated (d-Bus, IPC, ….) and human written statements (Status not received, initialized successfully, ….). The developers use the generated log for debugging. The entries have been configured to have a similar format depending on the generated system (timestamp, ids, component, context, verbosity level, description, ….).
1. The history of the identified anomalies is minimal and not archived.
2. Not many similar event templates in log files.
3. Software execution rules are not clearly documented.
4. The log events are co-related.
What are the recommended algorithms (Statistical, NLP, ML, Neural networks) that can be used to efficiently perform pattern extraction on the entries and identify existing and new anomalous behavior?