7
$\begingroup$

What loss function is most appropriate when training a model with target values that are probabilities? For example, I have a 3-output model. I want to train it with a feature vector $x=[x_1, x_2, \dots, x_N]$ and a target $y=[0.2, 0.3, 0.5]$.

It seems like something like cross-entropy doesn't make sense here since it assumes that a single target is the correct label.

Would something like MSE (after applying softmax) make sense, or is there a better loss function?

$\endgroup$

1 Answer 1

10
$\begingroup$

Actually, the cross-entropy loss function would be appropriate here, since it measures the "distance" between a distribution $q$ and the "true" distribution $p$.

You are right, though, that using a loss function called "cross_entropy" in many APIs would be a mistake. This is because these functions, as you said, assume a one-hot label. You would need to use the general cross-entropy function,

$$H(p,q)=-\sum_{x\in X} p(x) \log q(x).$$ $ $

Note that one-hot labels would mean that $$ p(x) = \begin{cases} 1 & \text{if }x \text{ is the true label}\\ 0 & \text{otherwise} \end{cases}$$

which causes the cross-entropy $H(p,q)$ to reduce to the form you're familiar with:

$$H(p,q) = -\log q(x_{label})$$

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .