1
$\begingroup$

We are given a computer vision classification task, that is, a task that asks us to predict the category of an image over $n$ predefined classes (the so-called closed set classification problem).

Question: Is it possible to give an estimate on what is the best accuracy one is likely achieve using an end-to-end CNN model (possibly, using a popular backbone) in this task? Do the performances of state-of-the-arts models on open datasets serve as a good reference? If someone claims that they achieve certain performance with some popular CNN architecture, how do we know s/he is not bragging?

You may or may not have access to the training dataset yet. The testing dataset shall be something close to the real-world production scenario. I know this is too vague, but just assume you have a fair judge.

Background: Product teams sometimes asks engineering teams for quick (and dirty) solutions. Engineering teams want to assess the feasibility before say "Yes we can do $95\%$" and officially launch (and be responsible) the projects.

Improve this question
$\endgroup$
1
  • $\begingroup$ If the task is similar, you can have an estimate of performance from the state-of-the-art. E.g. if you want to build a facial recognition CNN, you can more or less know what to expect. This assumes that your dataset somewhat follows those that the sota is benchmarked upon. In different tasks you have no chance at all to see what you are expecting. $\endgroup$
    – Djib2011
    Mar 31, 2020 at 0:13

1 Answer 1

Reset to default
2
$\begingroup$

There is no easy rule for this. You can use transfer learning to select a model that works well on image classification. However the accuracy you achieve will be highly dependent on your training set. If your training set is "similar" in quantity and quality to what was used for the accuracy achieved by the transfer learning model in some application you have a reasonable chance of coming close to that accuracy. By similar is mean roughly the same number of images per class. By quality I mean what percentage of the pixels in the images that are occupied by the "region of interest -ROI), same level of noise in the images etc. Also it depends on the nature of the classes. If the classes are widely different (elephants vs trees) the accuracy should be higher than if you are try to classify closely related images (human faces).

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks for the answer. What if we assume the classification problems comes from production requirements in a specific domain, so it looks more like a fine-grained classification? (like those in iMaterialist Challenges distinguishing fashion/retail products/furniture) In such cases, "the nature of the classes" is proposing more difficulties. Is it highly likely that we won't get better performance on a 1000-class task than those on ImageNet leaderboard with the same backbone? $\endgroup$
    – AlgRev
    Mar 31, 2020 at 14:29

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .