1
$\begingroup$

I'm evaluating the accuracy in detecting objects for my image data set using three deep learning algorithms. I have selected a sample of 30 images. To measure the accuracy, I manually count the number of objects in each image and then calculate recall and precision values for three algorithms. Following is a sample:

enter image description here

Finally to select the best model for my data set, can I calculate the mean Recall and mean Accuracy? For Example:

enter image description here

Improve this question
$\endgroup$
5
  • $\begingroup$ By precision do you mean IoU? Or any other else? $\endgroup$
    – Clement
    Jan 5, 2020 at 22:19
  • $\begingroup$ @Clement Hui : Precision is simply $\frac{TP}{PP} = \frac{TP}{TP+FN}$. I have counted manually the number of objects in each image. Then I obtain the count of objects by the deep learning model. $\endgroup$
    – Nilani Algiriyage
    Jan 5, 2020 at 22:26
  • $\begingroup$ ? What do you mean by TP PP and FN? Thanks $\endgroup$
    – Clement
    Jan 5, 2020 at 22:28
  • $\begingroup$ @Clement Hui: It is the number of true positives(TP) over the number of predicted positives (PP). (FN) is false negatives. Thanks. $\endgroup$
    – Nilani Algiriyage
    Jan 5, 2020 at 22:32
  • $\begingroup$ Oh ok. So isn't calculating the accuracy and precision just counting teh number of all of these and plug it in the function? Can't you do it using a program? Thanks $\endgroup$
    – Clement
    Jan 5, 2020 at 22:35

1 Answer 1

Reset to default
3
$\begingroup$

For the precision metric for example you have:

$$ Precision = \frac{TP}{TP+FP}, $$ with TP = True Positive and FP = False Positive.

Imagine you have the following values:
Image 1: $TP = 2, FP = 3$
Image 2: $TP = 1, FP = 4$
Image 3: $TP = 3, FP = 0$

The precision scores as you calculated will be:
Image 1: $2/5$
Image 2: $1/5$
Image 3: $1$
Your average will be: $0.533$

On the other hand if you sum them all up and then calculate the precision value you get:

$P = \frac{6}{6+7} = 0.462$

This proves that averaging the precision scores is not the same as calculating the total precision in one go.

Since what you want is to know how precise your algorithm is, independently of the precision for each image, you should sum all the TP and FP and only then calculate the precision for each model. This way you will not have a biased average. The average would give the same weight to an image with a larger number of objects as to another image which had fewer objects.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you very much for the suggestion. I think that works well than taking the mean. $\endgroup$
    – Nilani Algiriyage
    Jan 7, 2020 at 20:34
  • $\begingroup$ @NilaniAlgiriyage You should also be careful when deciding what metric you are going for. It may depend on how your data-set is distributed (from perfectly balanced to largely imbalanced). Besides, the task at hand and target for the client may differ. F1-score, MCC (Matthews correlation coefficient for binary classification) and ROC (for comparing models) are the ones you need to consider. $\endgroup$
    – bit_scientist
    Jan 10, 2020 at 0:58

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .