# Questions tagged [cross-entropy]

For questions related to the concept of cross-entropy in the context of artificial intelligence. For example, when the cross-entropy is used as a loss function to train a neural network.

30 questions
Filter by
Sorted by
Tagged with
0 votes
0 answers
15 views

### Deriving the cross entropy loss via maximum-likelihood estimation?

For multi-class classification problems, we use the cross entropy loss, which can be derived from a multinomial distribution via the maximum likelihoos estimation method. I've already tried to derive ...
• 121
1 vote
1 answer
33 views

### Trouble writing the backpropagation algorithm in python through crossentropy and softmax

so I am writing my own neural network library for a class project and I got everything working for a simple 2-class test using the distance (L2) cost function. I wanted to get a similar result using ...
0 votes
0 answers
319 views

### Focal Loss vs Weighted Cross Entropy Loss

Weighted Focal Loss is defined like so $FL(p_t) = -\alpha_t log(p_t) (1-p_t)^\gamma$ Whereas weighted Cross Entropy Loss is defined like so $CE(p_t) = -\alpha_t log(p_t)$ Some blog posts try to ...
• 669
-1 votes
1 answer
88 views

### Why is the cross-entropy a cost function?

The question looks foolish, but I think cross-entropy is somewhat weird as a cost function. As a cost function for linear regression, the mean square error $\sum_{i=1}^{n} (y_i - (ax_i+b)) ^2$ seems ...
• 125
0 votes
0 answers
39 views

### Is categorical cross entropy better than binary cross entropy for imbalanced binary classification problems

I am training a NN model. The data is highly imbalanced (3% for positive labels), and I have not resampled more true classes in the training set. The model performs much better when categorical cross-...
• 109
0 votes
0 answers
126 views

### ignoring instances or masking by zero in multitask learning model

For a multitask learning model, I've seen that approaches usually mask the output that doesn't have a label with zeros. As an example, have a look here: How to Multi-task learning with missing labels ...
• 123
3 votes
3 answers
1k views

• 405
3 votes
2 answers
206 views

### Where is the mistake in my derivation of the GAN loss function?

I was pondering on the loss function of GAN, and the following thing turned out \begin{aligned} L(D, G) & = \mathbb{E}_{x \sim p_{r}(x)} [\log D(x)] + \mathbb{E}_{x \sim p_g(x)} [\log(1 - D(x)] \...
• 174
2 votes
1 answer
209 views

### How does the implementation of the VAE's objective function equate to ELBO?

For a lot of VAE implementations I've seen in code, it's not really obvious to me how it equates to ELBO. $$L(X)=H(Q)-H(Q:P(X,Z))=\sum_ZQ(Z)logP(Z,X)-\sum_ZQ(Z)log(Q(Z))$$ The above is the definition ...
• 597
1 vote
1 answer
203 views

• 43
1 vote
1 answer
62 views

### Are sentences from the same document independent and identically distributed?

I am trying to build an LSTM model to generate Shakspeare-like poems. I have data set $\{s_1, s_2, \dots, s_m \}$, which are sentences of Shakespeare poems, and each sentence contains words \$\{w_1, ...
• 43
2 votes
1 answer
55 views

### How should I penalize the model proportionally to the error?

I am making an MNIST classifier. I am using categorical cross-entropy as my loss function. I want to make it so that if the correct label is 3, then it will penalize the model less heavily if it ...
1 vote
1 answer
78 views

### Why does PyTorch use a different formula for the cross-entropy?

In my understanding, the formula to calculate the cross-entropy is $$H(p,q) = - \sum p_i \log(q_i)$$ But in PyTorch nn.CrossEntropyLoss is calculated using this ...
• 2,541
3 votes
1 answer
111 views

### How to formalize learning in terms of information theory?

Consider the following game on a MNIST dataset: There are 60000 images. You can pick any 1000 images and train your Neural Network without access to the rest of images. Your final result is ...
• 131
4 votes
1 answer
382 views

### Why does the binary cross-entropy work better than categorical cross-entropy in a multi-class single label problem?

I was just doing a simple NN example with the fashion MNIST dataset, where I was getting 97% accuracy, when I noticed that I was using Binary cross-entropy instead of categorical cross-entropy by ...
7 votes
1 answer
7k views

### How is division by zero avoided when implementing back-propagation for a neural network with sigmoid at the output neuron?

I am building a neural network for which I am using the sigmoid function as the activation function for the single output neuron at the end. Since the sigmoid function is known to take any number and ...
• 73
2 votes
2 answers
287 views

### Why do non-linear activation functions that produce values larger than 1 or smaller than 0 work?

Why do non-linear activation functions that produce values larger than 1 or smaller than 0 work? My understanding is that neurons can only produce values between 0 and 1, and that this assumption can ...
18 votes
1 answer
4k views

### Why has the cross-entropy become the classification standard loss function and not Kullback-Leibler divergence?

The cross-entropy is identical to the KL divergence plus the entropy of the target distribution. The KL divergence equals zero when the two distributions are the same, which seems more intuitive to me ...
• 291
9 votes
2 answers
10k views

### How do I handle negative rewards in policy gradients with the cross-entropy loss function?

I am using policy gradients in my reinforcement learning algorithm, and occasionally my environment provides a severe penalty (i.e. negative reward) when a wrong move is made. I'm using a neural ...
• 199