# How to estimate the accuracy upper limit of any CNN model over a computer vision classification task

We are given a computer vision classification task, that is, a task that asks us to predict the category of an image over $$n$$ predefined classes (the so-called closed set classification problem).

Question: Is it possible to give an estimate on what is the best accuracy one is likely achieve using an end-to-end CNN model (possibly, using a popular backbone) in this task? Do the performances of state-of-the-arts models on open datasets serve as a good reference? If someone claims that they achieve certain performance with some popular CNN architecture, how do we know s/he is not bragging?

You may or may not have access to the training dataset yet. The testing dataset shall be something close to the real-world production scenario. I know this is too vague, but just assume you have a fair judge.

Background: Product teams sometimes asks engineering teams for quick (and dirty) solutions. Engineering teams want to assess the feasibility before say "Yes we can do $$95\%$$" and officially launch (and be responsible) the projects.

• If the task is similar, you can have an estimate of performance from the state-of-the-art. E.g. if you want to build a facial recognition CNN, you can more or less know what to expect. This assumes that your dataset somewhat follows those that the sota is benchmarked upon. In different tasks you have no chance at all to see what you are expecting. – Djib2011 Mar 31 '20 at 0:13