I am trying to generate a word vector representation of the textual descriptions of events from the log file in a distributed system. The logged events are time series data and correlated. During the pre-processing, I have removed the unwanted entries(columns representing timestamp, ids, log level, ...) and special characters in the descriptions part. I now need to convert the filtered textual data to a vector representation to perform feature extraction and clustering. The goal is to perform an Unsupervised Anomaly Detection from the log data. Would TF-IDF be the ideal choice in such a scenario? Will a sparse matrix impact the succeeding steps?

Note: The log file also contains vocabulary specific to the system which is not contained in the English dictionary.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.