# Formal definition of the Object Detection problem

For many problems in computer science, there is a formal, mathematical problem defition.
Something like: Given ..., the problem is to ...

How can the Object Detection problem (i.e. detecting objects on an image) be formally defined?

Given a set of pixels, the task is to decide

1. which pixels belong to an object at all,
2. which pixels belong to the same object.

How can this be put into a formula?

• For the first question, a typical way to formalize it might be defining a characteristic function: $$\chi_o (p) = \begin{cases} 1, & \text{if pixel p belongs to object o,}\\ 0, & \text{else } \end{cases}$$ – JavAlex Sep 25 at 15:25
• That formulation would be ok for image (or object) segmentation (where you need to classify individual pixels), but, in an object detection problem (which is a different problem than image segmentation), you do not need to classify individual pixels. You only need to find if the image contains an object of class $c$, where is that object (i.e. locate it), and maybe draw a bounding box around it. Maybe you are interested in image (or instance) segmentation. If that's the case, please, edit your post to say that. – nbro Sep 29 at 15:54
• Thank you, @nbro. I am interested in object detection, not image segmentation. I just thought that using pixel-based information could be a workaround to find a formula for object detection. Actually, image segmentation could be the first step of the object detection task: 1. classify single pixels and assign probabilities to them of belonging to a specific object, 2. merge pixels to objects. – JavAlex Sep 30 at 12:27
• The problem with that approach is that you need labelling information for all pixels, which may be expensive to acquire. That's why pixel-level classification to perform object detection may be overkill, but, of course, it's possible. In fact, image/instance segmentation can be thought of as a form of object detection (but a fine-grained one, let's say). – nbro Sep 30 at 12:30

This is just an idea

Given a set of pixels, the task is to decide:

1. Which pixel is the center of an object?
2. What is the size of the bounding boxes with the center is the pixel in part 1?

Formula, consider this is a 2D image, call $$(x,y)$$ is the horizontal and vertical coordinate and $$(w_i,h_i)$$ is the size of bouding box of object $$i$$:

$$\text{For }m \in[x,x+w_i] \text{ and } n\in[y,y+h_i]$$

$$c_i(m,n) = \begin{cases} 1, \text{if pixel at position (m,n) is belongs to object i,}\\ 0, \text{else} \end{cases}$$

• It's important to note that classifying pixels is known as image (or instance) segmentation, which is a different task than object detection (finding objects in images, maybe by providing the bounding box around the found objects). So, you do not need to classify individual pixels in the object detection task. You just need to find where the objects are (e.g. the center of the bounding box around them). – nbro Sep 29 at 15:49
• I knew what you mean, but the question of @JavAlex asked for the formula, so I thought about that equation based on his comment. – Toby Sep 30 at 4:03
• Thank you very much @Toby. That sounds like a good first idea – JavAlex Sep 30 at 12:28
• @JavAlex I just wonder, why you need to define object detection task into mathematics formula? – Toby Oct 2 at 5:07

The formula from the Toby is right, I just add some kinds of stuff that can be used to detect the pixel belongs to an object or not. I found the below formula from this link. We can ideally take $$alpha=1$$ and $$beta=0$$ but practically it lies between [0,1] interval

# Abstract view of the object detection process

The first step we need to do is identifying the pixels which belong to the background and which belong to the foreground. Why we do this. when we identify the background and remove the background from an image it is very easier to identify the desire object pixels in the foreground.

The second thing is Labelling pixels/ regions as foreground and background.

Basically, this detection process depends on two important factors and they are similarities and discontinuities of pixels.

What I mean by similarities is all the pixels or most of the pixels will have some similarity with them like the same colour range (yellow) or intensity. Some common techniques that are used to identify the similarity are thresholding, Region growing and region splitting.

What I mean by discontinuities is when we move from the pixels of the object to pixels of the background then we can see some sudden differences like edges. Some techniques are used like edge contour tracking where all edges are tracked, local analysis and edge linking and hough transform.

## Why normal Edge detection fails to detect objects

We can use discontinuities to achieve segmentation if that so, normal edge detection is enough but we could not do object detection with normal edge detection like canny edge detection why? Since most of the edge detection techniques include canny edge detection returns thick, noisy and discontinued edges.

so basically the $$object$$ $$boundary$$ must be completed one and thickness is ideally 1 pixel.

That's why we could not do segmentation with edge detection only. so we need to do some analysis/ further processing to detect boundaries. Basic techniques are contour tracking, local analysis and global analysis with hough transform.

The most crucial part of object detection is how you are labelling and it depends on your problem domain.

• cs.cmu.edu/~efros/courses/AP06/Papers/borenstein-eccv-2002.pdf – rcvaram Sep 29 at 14:59
• You may design a e.g. neural network to produce a probability that an image contains an object of class $c_j$, so you design a neural network to produce $p(c_j \mid x)$, where $x$ is the image and $c_j$ is the class of object $j$. Then, if the probability is greater than e.g. $0.5$ (i.e. if $p(c_j \mid x) > 0.5$), you classify $x$ as containing an object $j$. If this is what you want to say, your formulation is quite confusing. In your formulation, you're basically assigning a probability if the image contains an object of a certain class, but how do you know that's the case? – nbro Sep 29 at 15:37
• And why do you care about probabilities once you know the image contains a certain object? Also, you're talking about "pixels", but the question is about object detection and not image segmentation, although you could classify each pixel even in object detection, but that's not usually the case. – nbro Sep 29 at 15:45
• Yes, but your formulation says something different. You're defining the conditional probability by relying on the fact that you know that $x$ contains or not $c$, but you don't usually know this (in fact, that's what you want to find out in the first place). – nbro Sep 29 at 15:59
• Thank you very much, @rcvaram, for the detailed explanation. Distinguishing foreground and background pixels sounds more like a binarization task. But in this way, one could possibly calculate confidence values for every single pixel, and then merge the pixels to objects. Region growing, region splitting and the linked papers are a great hint. Thanks! – JavAlex Sep 30 at 12:32