I am working on a task that requires me to classify a large amount of mixed files on a backup drive (more than 10 TB with more than 32 million files) based on content. The included file types are documents, images, videos, executable, and pretty much everything in between.
I am also required to create new tags or metadata that will allow for automatic classification of new files. It'd also allow for manual input of category. For each input I give to the system, the system would learn and improve its classification.
Here is what I have come up with so far:
- Documents: classify using existing categories with packages like Nltk on Python. Alternatively, first run topic modeling using LDA or NMF and then classify.
- Images: use CNN. In case of unknown label, use VAE to cluster the images.
- Videos and other types of files: I do not know how to approach this.
Since I am not sure about my approach, any input is greatly appreciated.