From what aspect to measure the performance of an object detector?

I am on the hook to measure the prediction results of an object detector. I learned from some tutorials that when testing a trained object detector, for each object in the test image, the following information is provided:

    <object>
<name>date</name>
<pose>Unspecified</pose>
<truncated>0</truncated>
<difficult>0</difficult>
<bndbox>
<xmin>451</xmin>
<ymin>182</ymin>
<xmax>695</xmax>
<ymax>359</ymax>
</bndbox>
</object>


However, it is still unclear to me 1) how does these information is taken by the object detector to measure the accuracy, and 2) how does the "loss" is computed for this case. Is it something like a strict comparison? For instance, if for the object "date", I got the following outputs:

    <object>
<name>date</name>
<pose>Unspecified</pose>
<truncated>0</truncated>
<difficult>0</difficult>
<bndbox>
<xmin>461</xmin>  <---- different
<ymin>182</ymin>
<xmax>695</xmax>
<ymax>359</ymax>
</bndbox>
</object>


Then I will believe that my object detector made something wrong? Or they tolerant some small delta such that if the bounding box has a small drifting, then it's acceptable. But if the "label" is totally wrong, then that's wrong for sure?

This is like a "blackbox" to me and it would be great if someone can shed some lights on this. Thank you.

box1 : coordinates : confidence box2 : coordinates : confidence etc...