CNN for detecting not just the nature of the object, but position within image as well

Question

Consider a typical convolutional neural network like this example that recognizes 10 different kinds of objects from the CIFAR-10 dataset:

https://github.com/tflearn/tflearn/blob/master/examples/images/convnet_cifar10.py

""" Convolutional network applied to CIFAR-10 dataset classification task.

References:
    Learning Multiple Layers of Features from Tiny Images, A. Krizhevsky, 2009.

Links:
    [CIFAR-10 Dataset](https://www.cs.toronto.edu/~kriz/cifar.html)

"""
from __future__ import division, print_function, absolute_import

import tflearn
from tflearn.data_utils import shuffle, to_categorical
from tflearn.layers.core import input_data, dropout, fully_connected
from tflearn.layers.conv import conv_2d, max_pool_2d
from tflearn.layers.estimator import regression
from tflearn.data_preprocessing import ImagePreprocessing
from tflearn.data_augmentation import ImageAugmentation

# Data loading and preprocessing
from tflearn.datasets import cifar10
(X, Y), (X_test, Y_test) = cifar10.load_data()
X, Y = shuffle(X, Y)
Y = to_categorical(Y, 10)
Y_test = to_categorical(Y_test, 10)

# Real-time data preprocessing
img_prep = ImagePreprocessing()
img_prep.add_featurewise_zero_center()
img_prep.add_featurewise_stdnorm()

# Real-time data augmentation
img_aug = ImageAugmentation()
img_aug.add_random_flip_leftright()
img_aug.add_random_rotation(max_angle=25.)

# Convolutional network building
network = input_data(shape=[None, 32, 32, 3],
                     data_preprocessing=img_prep,
                     data_augmentation=img_aug)
network = conv_2d(network, 32, 3, activation='relu')
network = max_pool_2d(network, 2)
network = conv_2d(network, 64, 3, activation='relu')
network = conv_2d(network, 64, 3, activation='relu')
network = max_pool_2d(network, 2)
network = fully_connected(network, 512, activation='relu')
network = dropout(network, 0.5)
network = fully_connected(network, 10, activation='softmax')
network = regression(network, optimizer='adam',
                     loss='categorical_crossentropy',
                     learning_rate=0.001)

# Train using classifier
model = tflearn.DNN(network, tensorboard_verbose=0)
model.fit(X, Y, n_epoch=50, shuffle=True, validation_set=(X_test, Y_test),
          show_metric=True, batch_size=96, run_id='cifar10_cnn')

It's a CNN with several layers, ending with 10 outputs, one for each type of object recognized.

But now think of a slightly different problem: Let's say I only want to recognize one type of object, but also detect its position within the image frame. Let's say I want to distinguish between:

• object is in center
• object is left of center
• object is right of center
• no recognizable object

Assume I build a CNN exactly like the one in the CIFAR-10 example, but only with 3 outputs:

• center
• left
• right

And of course, if none of the outputs fires, then there is no recognizable object.

Assume I have a large training corpus of images, with the same kind of object in many different positions within the image, the set is grouped and annotated properly, and I train the CNN using the usual methods.

Should I expect the CNN to just "magically" work? Or are there different kinds of architectures required to deal with object position? If so, what are those architectures?

Answer 1

You could use another type of CNN that instead of classification is performing regression so it will also give you as output the position(it's not really like that but this is the core idea) . Some algorithms are SSD or YOLO.

Answer 2

I guess one of the simplest approach would be train CNN to detect the object in a given image i.e the CNN has single output whole value indicates the probability of the object being in image and then just apply the CNN by segmenting the image into the desired sections and selecting the section which has the highest and good enough probability. For better results I would suggest to train the CNN on the object images with very less other information aka other objects in the images.

Answer 3

A simple trick can be splitting the image in to three frames vertically and feeding them to the image net and you can decide the position by looking for the frame which has higher probability of the desired category(simply max of all the probs). Or else you can try YOLO algorithm which further uses non max suppression and IOU on the frames.

Answer 4

Object detection models work in a very similar fashion to what you have proposed. They output dense predictions at reduced resolutions. Each prediction fires if an object center is located within the respective region of the image. Of course, there are various further developments, but the main idea is exactly that.

Answer 5

One of the suggestions in the accepted answer was SSD. On their website, SSD mentioned a competitor, faster_rcnn. faster_rcnn was deprecated in favor of Detectron. Detectron was deprecated in favor of Detectron2. Long live detectron2.

It looks pretty cool and powerful: https://github.com/facebookresearch/detectron2