From what aspect to measure the performance of an object detector?

Question

I am on the hook to measure the prediction results of an object detector. I learned from some tutorials that when testing a trained object detector, for each object in the test image, the following information is provided:

    <object>
    <name>date</name>
    <pose>Unspecified</pose>
    <truncated>0</truncated>
    <difficult>0</difficult>
    <bndbox>
        <xmin>451</xmin>
        <ymin>182</ymin>
        <xmax>695</xmax>
        <ymax>359</ymax>
    </bndbox>
</object>

However, it is still unclear to me 1) how does these information is taken by the object detector to measure the accuracy, and 2) how does the "loss" is computed for this case. Is it something like a strict comparison? For instance, if for the object "date", I got the following outputs:

    <object>
    <name>date</name>
    <pose>Unspecified</pose>
    <truncated>0</truncated>
    <difficult>0</difficult>
    <bndbox>
        <xmin>461</xmin>  <---- different
        <ymin>182</ymin>
        <xmax>695</xmax>
        <ymax>359</ymax>
    </bndbox>
</object>

Then I will believe that my object detector made something wrong? Or they tolerant some small delta such that if the bounding box has a small drifting, then it's acceptable. But if the "label" is totally wrong, then that's wrong for sure?

This is like a "blackbox" to me and it would be great if someone can shed some lights on this. Thank you.

Answer 1

This is a high level explanation: So most object detectors are generally deep neural networks that return a set of boxes:

box1 : coordinates : confidence box2 : coordinates : confidence etc...

and loss is computed different model to model. But i think your specifically curious how most do box comparison and loss. Generally they first check if the box is even relevant to the target. To do this often they calculate the Intersection over Union (IOU) as known as the Jaccard index wikipedia link. This checks how overlapped they are biased on the box sizes. Now if its above some threshold (example: boxes have IOU over .5 caluclate a loss, otherwise loss is just 0) they will compute some objective. Objectives can differ, but ideally its something differentiable. Examples include vectorizing and approximiating the IOU, dice loss, etc..

After a quick google search, heres a good tutorial explaining some components commonly used in the field (region proposals, feature extraction, etc) [guide]