# Questions tagged [cross-entropy]

For questions related to the concept of cross-entropy in the context of artificial intelligence. For example, when the cross-entropy is used as a loss function to train a neural network.

23 questions
Filter by
Sorted by
Tagged with
64 views

146 views

### Mathematical Analysis of the Loss function of GAN

I was pondering on loss function of GAN and following thing turned out \begin{aligned} L(D, G) & = \mathbb{E}_{x \sim p_{r}(x)} [\log D(x)] + \mathbb{E}_{x \sim p_g(x)} [\log(1 - D(x)] \\ & =...
60 views

### How does implementation of VAE's objective function equate to ELBO?

For a lot of VAE implementations I've seen in code, it's not really obvious to me how it equates to ELBO. $$L(X)=H(Q)-H(Q:P(X,Z))=\sum_ZQ(Z)logP(Z,X)-\sum_ZQ(Z)log(Q(Z))$$ The above is the definition ...
43 views

### Understanding the MuZero loss function for a two-player game

This question is connected to a question that I asked some time ago. This is how I understood the training procedure takes place (please correct any conceptual mistakes here): Many complete games are ...
180 views

58 views

### Are sentences from the same document independent and identically distributed?

I am trying to build an LSTM model to generate Shakspeare-like poems. I have data set $\{s_1, s_2, \dots, s_m \}$, which are sentences of Shakespeare poems, and each sentence contains words \$\{w_1, ...
52 views

### How should I penalize the model proportionally to the error?

I am making an MNIST classifier. I am using categorical cross-entropy as my loss function. I want to make it so that if the correct label is 3, then it will penalize the model less heavily if it ...
60 views

### Why does PyTorch use a different formula for the cross-entropy?

In my understanding, the formula to calculate the cross-entropy is $$H(p,q) = - \sum p_i \log(q_i)$$ But in PyTorch nn.CrossEntropyLoss is calculated using this ...
103 views

### How to formalize learning in terms of information theory?

Consider the following game on a MNIST dataset: There are 60000 images. You can pick any 1000 images and train your Neural Network without access to the rest of images. Your final result is ...
306 views

### Why does the binary cross-entropy work better than categorical cross-entropy in a multi-class single label problem?

I was just doing a simple NN example with the fashion MNIST dataset, where I was getting 97% accuracy, when I noticed that I was using Binary cross-entropy instead of categorical cross-entropy by ...