What are the main algorithms used in computer vision?

Question

Nowadays, CV has really achieved great performance in many different areas. However, it is not clear what a CV algorithm is.

What are some examples of CV algorithms that are commonly used nowadays and have achieved state-of-the-art performance?

Answer 1

There are many computer vision (CV) algorithms and models that are used for different purposes. So, of course, I cannot list all of them, but I can enumerate some of them based on my experience and knowledge. Of course, this answer will only give you a flavor of the type of algorithm or model that you will find while solving CV tasks.

For example, there are algorithms that are used to extract keypoints and descriptors (which are often collectively called features, although the descriptor is the actual feature vector and the keypoint is the actual feature, and in deep learning this distinction between keypoints and descriptors does not even exist, AFAIK) from images, i.e. feature extraction algorithms, such as SIFT, BRISK, FREAK, SURF or ORB. There are also edge and corner detectors. For example, the Harris corner detector is a very famous corner detector.

Nowadays, convolutional neural networks (CNNs) have basically supplanted all these algorithms in many cases, especially when enough data is available. Rather than extracting the typical features from an image (such as corners), CNNs extract features that are most useful to solve the task that you want to solve by taking into account the information in the training data (which probably includes corners too!). Hence CNNs are often called data-driven feature extractors. There are different types of CNNs. For example, CNNs that were designed for semantic segmentation (which is a CV task/problem), such as the u-net, or CNNs that were designed for instance segmentation, such as mask R-CNN.

There are also algorithms that can be used to normalize features, such as the bag-of-features algorithm, which can be used to create fixed-size feature vectors. This can be particularly useful for tasks like content-based image retrieval.

There are many other algorithms that could be considered CV algorithms or are used to solve CV tasks. For example, RanSaC, which is a very general algorithm to fit models to data in the presence of outliers, can be used to fit homographies (matrices that are generally used to transform planes to other planes) that transform pixels of one image to another coordinate system of another image. This can be useful for the purpose of template matching (which is another CV task), where you want to find a template image in another target image. This is very similar to object detection.

There are also many image processing algorithms and techniques that are heavily used in computer vision. For example, all the filters (such as Gaussian, median, bilateral, non-local means, etc.) that can be used to smooth, blur or de-noise images. Nowadays, some deep learning techniques have also replaced some of these filters and image processing techniques, such as de-noising auto-encoders.

All these algorithms and models have something in common: they are used to process images and/or get low- or high-level information from images. Most of them are typically used to extract features (i.e. regions of the images that are relevant in some way) from images, so that they can later be used to train a classifier or regressor to perform some kind of task (e.g. find and distinguish the objects, such people, cars, dogs, etc. in an image). The classifiers/regressors are typically machine learning (ML) models, such as SVMs or fully-connected neural networks, but there's a high degree of overlap between CV and ML because some ML tools are used to solve CV tasks (e.g. image classification).

Answer 2

Computer vision is a wide field, and besides the fact that deep learning dominates, there are still many, many other algorithms that see widespread use in both academia and industry.

For tasks such as image classification / object recognition, the typical paradigm is some CNN architecture such as a ResNet or VGG. There has been lots of works to extend and improves the CNNs, but the basic architecture has not really changed much over the years. Interestingly, there's been some work to encode more complex inductive biases / invariants into the deep learning modelling process, such as Spatial Transformer Networks and Group Equivariant Networks. More classical vision approaches to such problems typically include computing some form of hand-crafted feature (HOG, LBP), and training any off-the-shelf classifier.

For object detection, the de-facto for many years was Viola-Jones for it's combination of performance and speed (even though there were more accurate systems at the time, but they were slower). More recently, object detection has been dominated by deep learning, with architectures such as SSD, YOLO, all the RCNN variants, etc.

A related problem to object detection is segmentation. Deep learning again dominates in this area with algorithms such as Mask RCNN. However, many other approaches exist and see some use, such as superpixels (e.g. SLIC), watershed, and normalized cuts.

For problems such as image search, vision approaches such as Fisher vectors and VLAD (computed from image descriptors such as SIFT or SURF) are still competitive. However, CNN features have also seen use in this domain.

For video analysis, CNNs (typically, 3D CNNs) are popular. However, they often leverage other vision techniques such as optical flow. The most popular optical flow algorithms are Brox, TVL-1, KLT, and Farneback. There are more recent approaches which attempt to use deep learning to actually learn the optical flow, though.

An overarching set of techniques that has so many varying applications are interest point detectors, image descriptors, and feature encoding techniques. Interest point detectors attempt to localise interest points in an image or video, and popular detectors include Harris, FAST, and MSER. Image descriptors are used to describe those interest points. Example descriptors include SIFT, SURF, KAZE, and ORB. The descriptors themselves can be used to do various things such as estimate homographies using the RANSAC algorithm (for applications such as panorama and camera stabilisation). However, the descriptors can also be encoded and pooled into a single fixed-length feature vector, which serves as the representation of the image. The most common approach to this encoding is bag of feature / bag of visual words. This is based on K-means. However, popular extensions / variants include Fisher vectors and VLAD.

Self-supervised and semi-supervised learning is also very popular nowadays in academia, and seeks to get the most of out the abundant unlabelled data. In a computer vision context, popular techniques include MoCo and SimCLR, but new methods are released almost weekly!

Another problem domain in computer vision is the ability to generate / synthesize images. The is not unique to computer vision, but the common algorithms for this are variational autoencoders (VAEs) and generative adversarial networks (GANs).