How to verify classification model trained on classification dataset on a detection dataset for classification purpose?

I am working on a problem that involves two tasks - detection and classification. There is no single dataset for both tasks. I am training two models, separate on detection dataset and another on classification dataset. I use the images from the detection dataset as input and get classification predictions on top of detected bounding boxes.

Dataset description :

1. Classification - Image of the single object (E.g. Car) in the center with a classification label.
2. Detection - Image with multiple objects (E.g. 4 Cars) with bounding box annotations.

Task - Detect objects(e.g. cars) from detection datasets and classify them into various categories.

How do I verify whether the classification model trained on the classification dataset is working on images from detection dataset? (In terms of classification accuracy)

I cannot manually label the images from the detection dataset for individual class labels. (Need expert domain knowledge)

How do I verify my classification model?

Is there any technique to do this ? Like domain transfer or any weakly-supervised method ?

• Can you please add instances from datasets to understand your problem better? – dasmehdix Nov 10 '20 at 12:38
• This very confusing to read. Why do you want to verify that the classifier is working on data from the detection dataset. Aren't you the one who decides what data is being used? – Recessive Nov 13 '20 at 1:09
• Did you try the YOLO model? – Aray Karjauv Nov 13 '20 at 11:50
• @Recessive Yes, it is confusing. Here's an example - 1. I have a detection dataset having annotations for cars from top view, I can detect cars but I want to classify them as well (Merc, BMW, etc.) 2. So, I train a classification model on a different dataset which has such labels. That dataset has single car image and labels. – Aniket Velhankar Nov 14 '20 at 5:23
• 3. When I apply this classifier on the detection dataset, I take individual detected cars and classify them but since the detection and classification dataset is different, I want to find how accurate my classification model is classifying cars in the detection dataset and not on the dataset I trained it on. For this, I need manual annotation in the detection dataset, which I can't do. So, I am in search of a solution. – Aniket Velhankar Nov 14 '20 at 5:24

The Problem

We can see from the question that existing information on detection and classification in the small automotive vehicle domain has been located (in the form of two independent sets of vectors usable for machine training), and there is no already existing mapping or other correspondence between the elements of one set and the elements of the other. They were obtained independently, remain independent, and are linked only by the conventions of the domain (today's aesthetically acceptable and thermodynamically workable forms of small vehicles).

The goal stated in the question is to create a computer vision system that both detects cars and classifies them leveraging the information contained in the two distinct sets.

In the vision systems of mammals, there are also two distinct equivalences of sets; one arising from a genetic algorithm, the DNA that is expressed during the formation of the neural net geometry and bio-electro-chemistry of the visual system in early development; and the cognitive and coordinative pathways in the cerebrum and cerebellum.

If a robot, wheelchair, or other vehicle is to avoid traffic, we must produce a system that in some way matches or exceeds the collision avoidance performance of mammals. In crime prevention, toll collection, sales lot inventory, county traffic analysis, and other like applications, performance will again be expected to match or exceed the performance of biological systems. If a person can record the make, model, year, color, and license plate strings, so should the machine we employ in these capacities.

Consequently, this question is pertinent beyond academic curiosity, as it is applicable in current research and development of products.

That this question author notices the lack of a unified data set that can be used to train it to detect and characterize in a single network objects of interest is apropos and key to the challenge of finding a solution.

Approach The simplest approach would be to compose the system of two functions.

1. $$\quad\mathcal{D}: \mathbb{I}^4 \to {(\mathbb{I}^2, \mathbb{I}^2)}_1, \; {(\mathbb{I}^2, \mathbb{I}^2)}_2, \; ...$$
2. $$\quad\mathcal{C}: {(\mathbb{I}^2, \mathbb{I}^2)}_i \to {(\mathbb{I})}_i$$

The four dimensions of input for $$\mathcal{D}$$, the detector, are horizontal position, vertical position, rgb index, and brightness to decribe the pixelized image; and the output are bounding boxes as two "corner" coordinates corresponding to each identified vehicle, the second coordinate being either relative to the first or to a specific corner of the entire frame. The categorizer, $$\mathcal{C}$$, receives as input bounding boxes and produces as output the index or code that maps to the categories corresponding to the labels of the training set available for categorization. The system can then be described as follows.

$$\quad\quad\mathcal{S}: \mathcal{C} \circ \mathcal{D}$$

If the system is not color, subtract one from the above dimensionality of the input. If the system processes video, add one to the dimensionality of the input and consider using LSTM or GRU cell types.

The above substitution represented by "$$\circ$$" appears to be what is meant by, "I use the images from the detection dataset as input and get classification predictions on top of detected bounding boxes."

The interrogative, "How do I verify whether the classification model trained on the classification dataset is working on images from detection dataset? (In terms of classification accuracy)," appears to refer to the fact that labels do not exist for the second set that correspond to input elements of the first set, so an accuracy metric cannot be directly obtained. Since there is no obvious automatic way of generating labels for the vehicles in the pre-detected images containing potentially multiple vehicles, there is no way to check actual results against expected results. Composing multiple vehicle images from the categorization set to use as test input to the entire system $$\mathcal{S}$$ will only be useful in evaluating an aspect of the performance of $$\mathcal{D}$$, not $$\mathcal{C}$$.

Solution

The only way to evaluate the accuracy and reliability of $$\mathcal{C}$$ is with portions of the set used to train it that were excluded from the training and trust that the vehicles depicted in those images were sufficiently representative of the concept "car" to provide consistency of accuracy and reliability across the range of those detected by $$\mathcal{D}$$ in the application of $$\mathcal{S}$$. This means that the leveraging of the information, even if optimized to the degree possible by any arbitrary algorithm or parallelism in the set of all possible algorithms or parallelisms, is limited by the categorization training set. The number of set elements and the comprehensiveness and distribution of categories within that set must be sufficient to achieve an approximate equality between these two accuracy metrics.

1. Categorizing a test sample from the labeled set for $$\mathcal{C}$$ excluded from the training
2. Categorizing the vehicles isolated by $$\mathcal{D}$$ from its training input

Of course this discussion is in a particular environment, that of the system defined as the two artificial networks, one involving convolution based recognition and the other involving feature extraction, and the two training sets. What is needed is a wider environment where known vehicles are in view so that performance data of $$\mathcal{S}$$ is evaluated and a tap on the transfer of information between $$\mathcal{D}$$ and $$\mathcal{C}$$ can be used to differentiate between mistakes made on either side of the tap point.

Unsupervised Approach

Another course of action could be to not use the training set for categorization on the training of $$\mathcal{C}$$ at all, but rather use feature extraction and auto-correlation in an "unsupervised" approach, and then evaluate the results of on the basis of the final convergence metrics at the point when stability in categorization is detected. In this case, the images in the bounding boxes output by $$\mathcal{D}$$ would be used as training data.

The auto-trained network realizing $$\mathcal{C}$$ can then be further evaluated using the entire categorization training set.

Further Research

Hybrids of these two approaches are possible. Also, the independent training only in the rarest of cases leads to optimal performance. Understanding feedback as originally treated with rigor by MacColl in chapter 8 of his Fundamental Theory of Servomechanisms, later applied to the problem of linearity and stability of analog circuitry, and then to training, first in the case of GANs, may lead to effective methods to bi-train the two networks.

That evolved biological networks are trained in situ is an indicator that the most optimal performance can be gained by finding training architectures and information flow strategies that create optimality in both components simultaneously. No biological niche has ever been filled by a neural component that is first optimized and then inserted or copied in some way to a larger brain system. That is no proof that such component-ware can be optimal, but there is also no proof that the DNA driven systems that have emerged are not nearly optimized for the majority of terrestrial conditions.

• This is the perfect explanation that I could have got! Can you elaborate on the "unsupervised" approach? Any papers/articles that are using the mentioned technique? – Aniket Velhankar Nov 15 '20 at 13:34

To verify the accuracy of the classification stage, you will need labeled images with a single car.

To train and verify accuracy of the detection stage and full system, you can:

1. in the datasets with images with multiple cars, manually, mark the image rectangles that contains one car.
2. from previous, split the image in one or more ones, each one containing a single car.
3. pass each one of the previous image with a single car to the classification stage (that means assume classification has 100% accuracy). Record its outputs (labeled cars).
4. now, from output of steps 1) and 3), you can produce labeled images with multiple cars. Use it to train the detector and verify full system accuracy.
• Yes, this is an interesting thought. But if I were to publish these results in a journal/conference using the said assumption, is it valid ? – Aniket Velhankar Nov 15 '20 at 13:17
• In any paper you must explain your methodology. – pasaba por aqui Nov 15 '20 at 18:51