1
$\begingroup$

I'm evaluating the accuracy in detecting objects for my image data set using three deep learning algorithms. I have selected a sample of 30 images. To measure the accuracy, I manually count the number of objects in each image and then calculate recall and precision values for three algorithms. Following is a sample:

enter image description here

Finally to select the best model for my data set, can I calculate the mean Recall and mean Accuracy? For Example:

enter image description here

$\endgroup$
  • $\begingroup$ By precision do you mean IoU? Or any other else? $\endgroup$ – Clement Hui Jan 5 at 22:19
  • $\begingroup$ @Clement Hui : Precision is simply $\frac{TP}{PP} = \frac{TP}{TP+FN}$. I have counted manually the number of objects in each image. Then I obtain the count of objects by the deep learning model. $\endgroup$ – Nilani Algiriyage Jan 5 at 22:26
  • $\begingroup$ ? What do you mean by TP PP and FN? Thanks $\endgroup$ – Clement Hui Jan 5 at 22:28
  • $\begingroup$ @Clement Hui: It is the number of true positives(TP) over the number of predicted positives (PP). (FN) is false negatives. Thanks. $\endgroup$ – Nilani Algiriyage Jan 5 at 22:32
  • $\begingroup$ Oh ok. So isn't calculating the accuracy and precision just counting teh number of all of these and plug it in the function? Can't you do it using a program? Thanks $\endgroup$ – Clement Hui Jan 5 at 22:35
3
$\begingroup$

For the precision metric for example you have:

$$ Precision = \frac{TP}{TP+FP}, $$ with TP = True Positive and FP = False Positive.

Imagine you have the following values:
Image 1: $TP = 2, FP = 3$
Image 2: $TP = 1, FP = 4$
Image 3: $TP = 3, FP = 0$

The precision scores as you calculated will be:
Image 1: $2/5$
Image 2: $1/5$
Image 3: $1$
Your average will be: $0.533$

On the other hand if you sum them all up and then calculate the precision value you get:

$P = \frac{6}{6+7} = 0.462$

This proves that averaging the precision scores is not the same as calculating the total precision in one go.

Since what you want is to know how precise your algorithm is, independently of the precision for each image, you should sum all the TP and FP and only then calculate the precision for each model. This way you will not have a biased average. The average would give the same weight to an image with a larger number of objects as to another image which had fewer objects.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thank you very much for the suggestion. I think that works well than taking the mean. $\endgroup$ – Nilani Algiriyage Jan 7 at 20:34
  • $\begingroup$ @NilaniAlgiriyage You should also be careful when deciding what metric you are going for. It may depend on how your data-set is distributed (from perfectly balanced to largely imbalanced). Besides, the task at hand and target for the client may differ. F1-score, MCC (Matthews correlation coefficient for binary classification) and ROC (for comparing models) are the ones you need to consider. $\endgroup$ – bit_scientist Jan 10 at 0:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.