1
$\begingroup$

I have a model for binary classification. The target variable has the different number of labels (instances) in each sample. For example, a batch of size 2 with 2 and 3 instances and correspondingly with 2 and 3 labels (0 and 1):

y_true = np.array([[0., 1., -1., -1.],
                   [1., 0., 1., -1.]])

The maximal number of labels (instances) in a sample is equal to 4. -1 values are used as a mask. I created a function (in TensorFlow) that masks all -1 values and calculates the loss for each unmasked value and then the average of all losses:

def my_loss_fn(y_true, y_pred):
    mask = tf.cast(tf.math.not_equal(y_true, tf.constant(-1.)), tf.float32)
    y_true, y_pred = tf.expand_dims(y_true, axis=-1), tf.expand_dims(y_pred, axis=-1)
    bce = tf.keras.losses.BinaryCrossentropy(reduction='none')
    return tf.reduce_sum(tf.cast(bce(y_true, y_pred), tf.float32) * mask) / tf.reduce_sum(mask)

Is it a correct way mathematically? Should I add some weights depending on number of labels in a sample? Or calculate loss per sample (per row in my example) and then get the average? Should I mask unused labels at all? It looks like my model doesn't learn to classify correctly and predicts 1 for all labels.

Improve this question
$\endgroup$
2
  • $\begingroup$ Correct for what model? The ordinary categorical cross-entropy loss arises from a specific probability model of the data. The categorical cross-entropy loss is the "correct" loss for that probability model, but it's "incorrect" if the data are assumed to arise from a different probability model. This reads a bit like an XY Problem. What problem are you trying to solve, and how does having a variable number of classes for each instance fit into solving it? $\endgroup$
    – Sycorax
    Commented Aug 10, 2022 at 18:06
  • $\begingroup$ Here's one possible solution. For a single instance, the categorical cross-entropy loss for $K$ classes is $$ L=-\sum_{k=1}^K y_k \log p_k=0 + \dots + 0 - \log p_t + 0 + \dots + 0 $$ where $y_k$ is the one-hot class label and $p_k$ is the estimated probability that the instance belongs to class $k$. Because $y_k$ is one-hot, there's only one nonzero term in the sum. The only part of this that involves the number of classes $K$ is the fact that there are $K-1$ zero entries in the sum. If this is the model you want, then no weighting or other refinement is needed! $\endgroup$
    – Sycorax
    Commented Aug 10, 2022 at 18:06

0

Reset to default

You must log in to answer this question.