Skip to main content
Share Your Experience: Take the 2024 Developer Survey
16 votes
Accepted

Why has the cross-entropy become the classification standard loss function and not Kullback-Leibler divergence?

When it comes to a classification problem in machine learning, the cross-entropy and the KL divergence are equal. As already stated in the question, the general formula is this: $$H(p, q) = H(p) + D_{...
Maxim's user avatar
  • 1,967
7 votes

How do I handle negative rewards in policy gradients with the cross-entropy loss function?

It depends on your loss function, but you probably need to tweak it. If you are using an update rule like loss = -log(probabilities) * reward, then your loss is ...
Tahlor's user avatar
  • 171
6 votes
Accepted

How does the implementation of the VAE's objective function equate to ELBO?

I don't want to think about the correctness of your supposed ELBO equation now. Nevertheless, it's true that the ELBO can be rewritten in different ways (e.g. if you expand the KL divergence below, by ...
nbro's user avatar
  • 40.8k
5 votes
Accepted

Why are there two versions of softmax cross entropy? Which one to use in what situation?

It's the same thing, first version is the special case of the more general one. In the first case you only have two classes, it's binary cross-entropy, and they also included iteration over batch of ...
Brale's user avatar
  • 2,406
4 votes
Accepted

Where is the mistake in my derivation of the GAN loss function?

I guess the issue is you lost track of where the samples came from and since you requested a math explanation I'll try to go step by step using my notation and without checking other material to avoid ...
Nicola Bernini's user avatar
3 votes

How do I handle negative rewards in policy gradients with the cross-entropy loss function?

The cross-entropy loss will always be positive because the probability is in the range $[0, 1]$, so $-ln(p)$ will always be positive.
user3711746's user avatar
3 votes

How to formalize learning in terms of information theory?

In short: It is easy to quantify information, but it is not easy to quantify its usefulness I'm not sure how exactly you are looking to formalise your experiment, but it might be helpful to consider ...
natanijelvasic's user avatar
3 votes
Accepted

Are sentences from the same document independent and identically distributed?

The sentences coming from the same document, author, etc., are unlikely to be independent, that is, the occurrence of a sentence $s_i$ in a certain document $d$ is likely correlated with the ...
nbro's user avatar
  • 40.8k
3 votes
Accepted

Which loss function should I use in REINFORCE, and what are the labels?

The loss function you are looking for is cross entropy loss. The 'label' that you use is the action you took at the time point you are updating for.
David's user avatar
  • 4,920
3 votes

In logistic regression, why is the binary cross-entropy loss function convex?

The $L_{CE}$ that you provided is binary cross-entropy, the factor $y$ and $(1-y)$ is because $y$ is binary $({0,1})$, careful with the name next time. The cross-entropy loss should have form: $$L_{CE}...
CuCaRot's user avatar
  • 912
2 votes

Why do non-linear activation functions that produce values larger than 1 or smaller than 0 work?

Why wouldn't they work? Each neuron's output is equal to a function over the sum of all its weights multiplied by their corresponding neurons. If that function is the Sigmoid function, then the output ...
BlueMoon93's user avatar
2 votes
Accepted

Why is the cross-entropy a cost function?

Optimizing the cross-entropy is equivalent to optimizing the log-likelihood of the parameters given the data, $\ell(\theta)$, which is what we want, i.e. find the parameters that most likely generated ...
nbro's user avatar
  • 40.8k
2 votes

What loss function will be correlated with classification metrics?

Different metrics measure different quantities, so there is no reason to expect different metrics to move together unless one is a function of the other (such as MSE and RMSE). Further, metrics like ...
Dave's user avatar
  • 618
2 votes

why cross entropy loss has to be multiplied by a batch size during an evaluation in transformer model?

That is because the default reduction for the CEP loss calculation is mean. Hence to find the true average across all batches, you first multiply by the batch size and then divide by total number of ...
Kushal Kumar's user avatar
1 vote

Backpropagation with CrossEntropy and Softmax, HOW?

The derivation of the softmax function is a bit tricky because except the other common activation functions(sigmoid, relu...) all the Zs values impact each other. It is because when you are ...
Ege's user avatar
  • 25
1 vote

Backpropagation with CrossEntropy and Softmax, HOW?

No it's not the same. First the derivative of the cost function is taken with respect to the weight and not the input. This is usually done using the chain rule of calculus. To calculate this using ...
Chiho's user avatar
  • 11
1 vote
Accepted

What is the correct loss function for binary classification: Cross entropy or Binary cross entropy?

Binary cross-entropy loss is a specific case for cross-entropy loss. Theoretically, one can also use the normal cross-entropy loss for binary classification. Binary cross-entropy is probably ...
Robin van Hoorn's user avatar
1 vote
Accepted

Trouble writing the backpropagation algorithm in python through crossentropy and softmax

I found the bug on my code, now everything works just fine, so I am fairly sure that the derivation of the formulas is on point. Optimisation wise, clearly using the short formula on the end of the ...
user605734 MBS's user avatar
1 vote

In logistic regression, why is the binary cross-entropy loss function convex?

I'm unable to comment on previous answers because I'm new to ai.stackexchange and don't have enough clout points. So I'm writing my comment as an answer instead. Unless I'm missing something, I ...
Chiraz BenAbdelkader's user avatar
1 vote

Where is the mistake in my derivation of the GAN loss function?

$\textbf{Remark.}$ I'd leave this as a comment if I could. Regarding notation (which I believe may be the cause of your issue here), the loss function is better written as \begin{align*} \operatorname{...
Scott Kirila's user avatar
1 vote

In logistic regression, why is the binary cross-entropy loss function convex?

If you find the Hessian matrix (the matrix of second order derivatives) for the binary cross entropy loss function, you'll see that it is positive semidefinite for any possible value of the parameters....
Abhijith S. Raj's user avatar
1 vote
Accepted

What error should I use for RNN?

To provide a good answer would fill several pages. To keep it very simple try many different loss functions on your model. Your goal is to have the highest performance based on some desired ...
Brian O'Donnell's user avatar
1 vote

How to compute the gradient of the cross-entropy loss function with respect to the parameters with softmax activation function?

Alright. Consider an ordinary neural network, so, in the last layer, we have, $z^{[L]} = W^{[L]} a^{[L-1]} + b^{[L]}$, where $a^{[L]} = \sigma(z^{[L]})$, where $\sigma$ is the softmax activation: $$ \...
kid's user avatar
  • 11
1 vote
Accepted

Have I understood the loss function from the original U-Net paper correctly?

Yes, $E$ is the cross-entropy function and a direct generalization of the binary case. For the binary case, probability to belong to the class $1$ is given by a sigmoid function $\sigma(x)$ of the ...
spiridon_the_sun_rotator's user avatar
1 vote

What is the advantage of using cross entropy loss & softmax?

Short answer: larger gradients That is not the derivative of the softmax function. $t - o$ is the combined derivative of the softmax function and cross entropy loss. Cross entropy loss is used to ...
S2673's user avatar
  • 590
1 vote

How do you manage negative rewards in policy gradients?

You don't need to manage negative rewards separately, if you implemented the algorithm correctly it will work regardless if the rewards are negative or not. You seem to be using rewards for the loss ...
Brale's user avatar
  • 2,406
1 vote
Accepted

How should I penalize the model proportionally to the error?

I want to make it so that if the correct label is 3, then it will penalize the model less heavily if it classifies a 4 than a 7 because 4 is closer numerically to 3 than 7 is. How do I do this? ...
Neil Slater's user avatar
  • 32.5k
1 vote
Accepted

Why does PyTorch use a different formula for the cross-entropy?

When you one-hot-encode your labels with $p_i \in \{0,1\}$ you get $p_i = 0$ iff $i$ is not correct and, equivalently, $p_i =1$ iff $i$ is correct. Hence, $p_i \log(q_i) = 0 \log(q_i) = 0 $ for all ...
Jonathan's user avatar
  • 304
1 vote

Why does the binary cross-entropy work better than categorical cross-entropy in a multi-class single label problem?

https://stats.stackexchange.com/questions/260505/machine-learning-should-i-use-a-categorical-cross-entropy-or-binary-cross-entro Is relevant. based on my reading when you have a NN and do Binary ...
Michael Hearn's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible