How does Pinterest decipher what's on unmarked pictures and categorize them?

Question

According to this article, Pinterest acquired VisualGraph, an image recognition and visual search technology startup.

How does Pinterest apply VisualGraph technology for machine vision, image recognition and visual search in order to classify the images?

In short, how do they predict the image categories? Based on what features?

Answer 1

One of the Pinterest's white paper about Human Curation and Convnets powering item-to-item recommendations^arxiv describes implementation of convolutional neural network (CNN) based visual features (VGG²⁰¹⁴, Faster R-CNN). This demonstrates the effectiveness of it (such image or object representations) which can improve user engagement. The visual features are computed using the process described in the previous study about visual search at Pinterest and can be used for more targeted features to be computed for related pins.

Here are the examples of detected visual objects from Pinterest's object detection pipeline:

^{Image source: Human Curation and Convnets: Powering Item-to-Item Recommendations on Pinterest, Page 4, Fig. 6}

The images are categorized by using dominant visual objects (individual objects seen in the image which passes a confidence threshold in Faster R-CNN) using fine-tuned VGG reranking variant. This allows Pinterest to introduce features such real-time recommendations for the users.

Check also this blog entry: Building a scalable machine vision pipeline.