I have a dataset of images belonging to $N$ classes, $A_1, A_2...A_n,B_1,B_2...B_m$ and I want to train a CNN to classify them. The classes can be considered as subclasses of two broader classes $A$ and $B$, therefore the confusion between $A_i$ and $A_j$ is much less problematic than the confusion between $A_i$ and $B_j$. Therefore I want the CNN to be trained in such a way that the difference between $A_i$ and $B_j$ is considered as more relevant.

1) Are there any loss functions that take this requirement into account? Could a weighted cross entropy work in this case?

2) How this loss would change if the classes were unbalanced?

Classic Question

Not only can the reliability or accuracy needs be asymmetrically distributed between category boundaries, but the asymmetry might not be describable in terms of a first degree polynomial translation of the error surface.

The question data set are images belonging to class $c \in \Big( A: \{A_1, \, ..., \, A_n\} \land B: \{B_1, \, ..., \, B_m\} \Big)$. The convolutional network must be trained with a sample of labeled images to categorize with a reliability of $1 - \delta$ and accuracy of $1 - \epsilon$, based on the PAC (probably approximately correct) framework.

How can the loss function reflect the conceptualization of optimal training result that the reliability of categorization between $A$ and $B$ be of greater value than categorization between $A_i$ and $A_j$ where $i \ne j$, and similarly with B.

Criteria X would state that the value of differentiation $D$ is not sub-category index dependent for differentiation between super-categories. That is, $\forall \; (i, j) \; \text{where} \; i \ne j$, $D(A_i, B_i) = D(A_i, B_j)$.

Criteria Y, the categorization WITHIN a super-category is not unlike the typical symmetric case $\forall \; (i, j, k) \; \text{where} \; i \ne j \land i \ne k \land j \ne k$, $D(A_i, A_j) = D(A_i, B_k)$.)

  1. (a) Are there any loss functions that take this requirement into account?

Let $1_p$ be the bit value that is 1 if and only if the label matches the super-category and let $1_b$ be the bit value that is 1 if and only if the label matches the super-category and the sub-category. Because of Criteria X, a conical and normalized loss function could then be defined as thus.

$\epsilon = \dfrac {(1 - 1_p) + \alpha (1 - 1_b)} {1 + \alpha}$

The proportional value of sub-category reliability in relation to super-category reliability $\alpha < 1.0$.

Because of criteria Y, there is no meaning to concavity in this context. Applying mean squaring to the loss simply changes the curvature and meaning of $\alpha$ in relation to the loss as can be seen here.

$\epsilon = \dfrac {(1 - 1_p)^2 + \alpha^2 (1 - 1_b)^2} {1 + \alpha^2} = \dfrac {(1 - 1_p) + \tau (1 - 1_b)} {1 + \tau}\text{, where }\tau = \alpha^2$

  1. (b) Could a weighted cross entropy work in this case?

Cross entropy between the two levels of categorization would be baseless unless there were correlative effects between super classification feature and sub classification feature inherent in the target concept of the learning.

  1. How this loss would change if the classes were unbalanced?

Although there may be schemes for modifying the loss function to compensate for skewed distribution of categories represented in the sample, this is primarily a data input problem best solved by improving data collection or choosing training algorithms, designs, architectures, treating input, and properly controlling hyper-parameters. The loss function represents the inverse of what is considered optimal, not how one achieves it.

  • can this function be minimized with stochastic gradient descent? – firion Oct 11 at 14:59

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.