0
$\begingroup$

I am trying to generate a word vector representation of the textual descriptions of events from the log file in a distributed system. The logged events are time series data and correlated. During the pre-processing, I have removed the unwanted entries(columns representing timestamp, ids, log level, ...) and special characters in the descriptions part. I now need to convert the filtered textual data to a vector representation to perform feature extraction and clustering. The goal is to perform an Unsupervised Anomaly Detection from the log data. Would TF-IDF be the ideal choice in such a scenario? Will a sparse matrix impact the succeeding steps?

Note: The log file also contains vocabulary specific to the system which is not contained in the English dictionary.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.