I'm reading through Regularizing Neural Networks By Penalizing Confident Output Distributions and I'm stuck on the equation in section 3.2. It's not clear to me at all that the self-entropy of the model output distribution $H(p_\theta(\mathbb{y}_i,\mathbb{x}))$, is the same as $D_{KL}(u||p_\theta(\mathbb{y}_i,\mathbb{x}))$.

I did try to reason through it as follows:

We know the penalised loss is:

$$ \begin{align} \mathcal{L}(\theta) & = - \sum_{i}\log p_{\theta}(\mathbb{y}_i|\mathbb{x}) - \beta H(p_{\theta}(\mathbb{y_i}|\mathbb{x})) \\ & = - \sum_{j=i}\log p_{\theta}(\mathbb{y}_i|\mathbb{x}) - \beta \sum_{i}p_{\theta}(\mathbb{y_i}|\mathbb{x})\log(p_{\theta}(\mathbb{y_i}|\mathbb{x})) \end{align} $$

and if we only care about minimizing the loss, we can feel free to add a constant term to it, like $\sum_{i}p_{\theta}(\mathbb{y_i}|\mathbb{x})\log(u)$, where $u = 1/K$ (I believe this is constant, because the $u$ comes out of the sum then the sum just adds to 1). It doesn't mean the RHS is the same as before. It just differs by a constant. So by doing this we get some new loss which is just off by a constant:

$$ \begin{align} \mathcal{L}'(\theta) & = - \sum_{j=i}\log p_{\theta}(\mathbb{y}_i|\mathbb{x}) - \beta \sum_{i}(p_{\theta}(\mathbb{y_i}|\mathbb{x})\log(p_{\theta}(\mathbb{y_i}|\mathbb{x})) + p_{\theta}(\mathbb{y_i}|\mathbb{x})\log(u)) \\ & = - \sum_{j=i}\log p_{\theta}(\mathbb{y_i}|\mathbb{x}) - \beta \sum_{i}(p_{\theta}(\mathbb{y_i}|\mathbb{x})\log(p_{\theta}(\mathbb{y_i}|\mathbb{x})/u)) \\ & = - \sum_{j=i}\log p_{\theta}(\mathbb{y_i}|\mathbb{x}) - \beta D_{KL}(p_{\theta}(\mathbb{y_i}|\mathbb{x})||u) \end{align} $$

Presuming I'm on the right track, I don't know a few things:

  • Where did the $\beta$ go?
  • Why are the $u$ and the $p_\theta$ swapped around?
  • Why are they then allowed to "reverse the direction of the KL divergence" to further their argument?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.