How to detect outlier images?

Question

Before I describe my challenge, I want to point out that I have searched extensively online for "outlier image detection", "anomaly images detection", etc., but all returned results are about finding anomalies or outliers within an image (e.g., a defect in a machined part, a tear in fabric, etc.).

This is not what I am looking for: I am looking to identify images (whole images) that depict an object that is different from the other images. For instance, I have a stream of images from a live camera. Each image depicts a single dog (different breeds) and each image is centered on the dog. A few images depict a cat, rather than a dog... I would like to detect these cat images (outliers).

This is not a simple classification problem, where I have a labeled data set of cats and dogs... because I don't want to define the 'outlier'. In other words, the approach must be adaptable to the data: if I were to suddenly add a few images of trash, the model must flag those trash images as novel/outliers because in comparison to the number of dog images, trash images (like cat images) are pretty rare (or new).

Would something like sequential k-means work for this problem? I thought about sequential k-means, since it operates on a data stream and will adapt to the number of clusters (i.e, k does not need to be pre-defined).

What other approaches would work? Any resources (e.g., links to tutorials, GitHub repositories, specific algorithms, methodologies) that might help?

Answer 1

Hi there @pookie you can approach this problem using unsupervised anomaly detection techniques. One such technique is to use an autoencoder neural network.

The idea is to train an autoencoder on a large set of dog images, and then use the trained network to encode (compress) each image into a lower-dimensional representation. The encoded representations should capture the key features of dog images. The network can then be used to reconstruct the original image from the encoded representation.

Now, when a new image (e.g., a cat image or a trash image) is presented, the network will not be able to reconstruct it well because it has not seen that kind of image before. The reconstruction error will be higher for the novel image, which can be used to identify it as an outlier.

You can use this approach to build a threshold-based anomaly detection system, where any image with a reconstruction error above a certain threshold is considered an outlier. Alternatively, you can use a more sophisticated anomaly detection algorithm like One-class SVM, Isolation Forest, or Local Outlier Factor.

Another option is to use clustering techniques like k-means or DBSCAN to cluster the dog images and identify clusters with very few points as outliers. However, this approach may not work well if the outlier images are visually similar to dog images.

Here are some resources that may help you get started:

• Anomaly Detection with Autoencoders using Keras in Python: machinecurve/anomalyDetectionAutoencoder-using-Keras
• One-class SVM for anomaly detection: scikit-learn/oneClassSVM
• Isolation Forest for anomaly detection: scikit-learn/isolationForest
• Local Outlier Factor for anomaly detection: scikit-learn/localOutlierFactor

If you found this helpful, don't forget to upvote :)

Answer 2

I agree with Triple S' answer, but as a preprocessing step I suggest to use some pre-trained image classification network without the last fully-connected layer. This will give you robust features which represent characteristics of the input image. But since these representations are high-dimensional, I'd apply an autoencoder to these features.

Actually I have implemented and blogged about such approach, but I'm not sure whether links to personal blogs are discouraged or not. Well at least each answer should work as a stand-alone resource. In my case the images originated from videos, and I wanted to cluster either the frames themselves or video clips.

I used this for mountain biking videos, which resulted in this kind of 2D map:

Each line represents a different mountain biking / downhill clip. Anyway, in your case I'd use 2D coordinates for outlier detection. They seem to contain enough interesting data, and are so easy to visualize as well!

Link to my blog article: Image and video clustering with an autoencoder. It also has examples on clustering frames from either Arnold Schwarzeneg­ger or James Bond films. It could group conceptually similar images together in the 2D plane, from different films.