2
$\begingroup$

I'm training an object detection model (SSD300) to detect and classify body poses in thermal images.

Even I have more than 2k different poses, but the background does not change much (I have only 5 different points of view).

I trained my model on these images (70% for the training and 30% for validation).

Now, I want to evaluate the model on an unbiased dataset.

Should I keep images of my dataset for this purpose or should I use a real life dataset ?

(A good solution would be to have a real life training set, but I don't have)

I tried both, but as expected, I have an mAP=0.9 when evaluated on similar pictures and mAP=0.5 when evaluated on completely different images.

Bonus question: is mAP a relevant metric when I want to show result to a client ? (e.g a client doesn't understand if I tell him "my model has a mAP=0.7")

Precision-Recall ? (but I have to choose a pose classification threshold...)

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

If you want to evaluate on real thermal image dataset, you can use this one.

Thermal Image dataset

is mAP a relevant metric when I want to show result to a client ? (e.g a client doesn't understand if I tell him "my model has a mAP=0.7")

Mean Average Precision is the relevant metric but it's more technical. You can start explaining with False Positives and False Negatives in the predictions. In turn, it leads to Precision and Recall. It mostly depends upon your use case because there will always be a trade-off between them.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Ty for your answer on the mAP but you did not answer my main question on "how to build a good evaluation/test set ?". $\endgroup$
    – antoine Mathu
    Mar 3, 2020 at 13:33
  • 1
    $\begingroup$ At the beginning itself, we have to split the data as train/val/test. It depends upon how much data you have for your task to train the model. The test set should be taken from almost the same data distribution of the training set. If we train a model with only daylight images and test with night conditions, it is a bad idea to judge the model. $\endgroup$
    – Sudhakar17
    Mar 3, 2020 at 15:04

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .