According to this article, Pinterest acquired VisualGraph, an image recognition and visual search technology startup.

How does Pinterest apply VisualGraph technology for machine vision, image recognition and visual search in order to classify the images?

In short, how do they predict the image categories? Based on what features?


One of the Pinterest's white paper about Human Curation and Convnets powering item-to-item recommendationsarxiv describes implementation of convolutional neural network (CNN) based visual features (VGG2014, Faster R-CNN). This demonstrates the effectiveness of it (such image or object representations) which can improve user engagement. The visual features are computed using the process described in the previous study about visual search at Pinterest and can be used for more targeted features to be computed for related pins.

Here are the examples of detected visual objects from Pinterest's object detection pipeline:

Fig. 6. Examples of detected visual objects from Pinterest’s object detection pipeline. Detection of objects allows for more targeted visual features to be computed for Related Pins

Image source: Human Curation and Convnets: Powering Item-to-Item Recommendations on Pinterest, Page 4, Fig. 6

The images are categorized by using dominant visual objects (individual objects seen in the image which passes a confidence threshold in Faster R-CNN) using fine-tuned VGG reranking variant. This allows Pinterest to introduce features such real-time recommendations for the users.

Check also this blog entry: Building a scalable machine vision pipeline.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.