4
$\begingroup$

I am new to the object recognition community. Here I am asking about the broadly accepted ways to calculate the error rate of a deep CNN when the network produces different results using the same data.

1. Problem introduction

Recently I was trying to replicate some classic deep CNNs for the object recognition tasks. Inputs are some 2D image data including objects and the output are the identification/classification results of the object. The implementation involves the use of Python and Keras.

The problem I was facing is that, I may get different validation results among multiple runs of the training even using the same training/validation data sets. To me, that made it hard to report the error rate of the model since every time the validation result may be different.

I think this difference is because of the randomness involved in different aspects of deep CNN, such as random initialization, the random ‘dropout’ used in the regulation, the ‘shuffle’ process used in the choosing of epochs, etc. But I do not know yet the “right” ways to deal with this difference when I want to calculate the error rate in object recognition field.

2. My exploration – online search

I have found some answers online here. The author proposed two ways, and he/she recommended the first one shown below:

The traditional and practical way to address this problem is to run your network many times (30+) and use statistics to summarize the performance of your model, and compare your model to other models.

The second way he/she introduced is to go to every relevant aspect of the deep CNN, to "freeze" their randomness nature on purpose. This kind of approach has also been introduced from Keras Q&A here. They call this issue the “making reproductive results”.

3. My exploration – in academia community (no result yet, need your help!)

Since I was not sure whether the two ways mentioned above are the “right” ones broadly accepted, I was going further exploring in the object recognition academia community.

Now I just begin to read from imageNet website. But I have not found the answer yet. Maybe you could help me knowing the answer easier. Thanks!

Daqi

$\endgroup$
0
$\begingroup$

Unless the entropy is collected from quantum, thermal, or other true entropy sources, it is considered pseudo-random at best. This is why you can have a deterministic process that is not perfectly repeatable, exhibiting some variance.

List of Sources of Pseudo-random Behavior

The list of sources in the question is a good start.

  • Initialization
  • Ddropout used in regulation
  • Mini-batch shuffle

If stochastic gradient descent is used in the back-propagation, that could be an additional source, and there may be others. It is difficult to say without scrutinizing the code. CDN emerged along different paths from CNN and MLP so there is no unambiguously distinct classical version of it.

The Right Way

Since the variance is a known phenominon and the norm, there may be no particular question for which the academic community needs to be consulted for an answer.

The right ways to deal with the non-repeatability of validation statistics related to object recognition also has a norm: Follow standard reporting conventions that apply to all experimental results reporting.

The only solution that is mathematically rigorous and not changing the experiment to produce a desired result (which is not science) is the first of the two found in your investigation.

  • The suggestion to perform $N$ runs where $N \ge 30$ and report mean and variance (not standard deviation) is reasonable and responsible.
  • The suggestion to do this with $O$ ADDITIONAL models where $O \ge 3$ is another such practice.

Freezing is Necessarily Arbitrary

Some would consider freezing pseudo-random sources arbitrarily something other than, "making reproductive results," and consider it fabricating reproducable results.

Unfortunately, the temptation to please the reader or corporate stakeholder occasionally preempts truth in reporting. The result is counterproductive. In the worst case, it leads to broad acceptance of the wrong way. Those in the academic community should know that, whether or not they unanimously do. Be a leader and do what is backed by solid reasoning, whether that is the first suggestion found or a third choice that does not involve modifying the experiment arbitrarily to force results to appear more exact than the algorithm, methodology, or framework actually is.

$\endgroup$
  • $\begingroup$ Thanks for the thoughtful answer, FelicityC. I see there are three approaches available: (1) Follow standard reporting conventions; (2) Using statistics to summarize the performance; (3) Freezing pseudo-random sources. For the 1st option, do you have any idea about what are the "standard conventions" applied in object recognition academia community? Any popular paper I may read, or any website, community I may touch? $\endgroup$ – Daqi Dong Oct 19 '18 at 17:01
  • $\begingroup$ When I wrote, "conventions that apply to all experimental results reporting," and, "mathematically rigorous and not changing the experiment to produce a desired result," I was speaking of the scientific method, hypothesis testing by unbiased experiment design and statistical treatment of results, not an AI specific standard. I recommend Thomas Kuhn's 'Scientific Revolutions', the source of the term Paradigm Shift. It discusses how those who have refused to dismiss anomalies are the ones that have furthered science and technology. Something that recurs is not an outlier. $\endgroup$ – FelicityC Oct 20 '18 at 2:42
  • $\begingroup$ I am running the same model multiple times (N == 30) determining the statistic status of the validation results. One question is that, why we set the lower bound of the amount of running time to 30? Is there any special tech or historical reason? $\endgroup$ – Daqi Dong Oct 26 '18 at 19:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.