15 votes
Accepted

Why has the cross-entropy become the classification standard loss function and not Kullback-Leibler divergence?

When it comes to a classification problem in machine learning, the cross-entropy and the KL divergence are equal. As already stated in the question, the general formula is this: $$H(p, q) = H(p) + D_{...
  • 1,897
6 votes

How do I handle negative rewards in policy gradients with the cross-entropy loss function?

It depends on your loss function, but you probably need to tweak it. If you are using an update rule like loss = -log(probabilities) * reward, then your loss is ...
  • 161
4 votes
Accepted

Where is the mistake in my derivation of the GAN loss function?

I guess the issue is you lost track of where the samples came from and since you requested a math explanation I'll try to go step by step using my notation and without checking other material to avoid ...
4 votes
Accepted

Why are there two versions of softmax cross entropy? Which one to use in what situation?

It's the same thing, first version is the special case of the more general one. In the first case you only have two classes, it's binary cross-entropy, and they also included iteration over batch of ...
  • 2,236
3 votes

In logistic regression, why is the binary cross-entropy loss function convex?

The $L_{CE}$ that you provided is binary cross-entropy, the factor $y$ and $(1-y)$ is because $y$ is binary $({0,1})$, careful with the name next time. The cross-entropy loss should have form: $$L_{CE}...
  • 857
3 votes
Accepted

Which loss function should I use in REINFORCE, and what are the labels?

The loss function you are looking for is cross entropy loss. The 'label' that you use is the action you took at the time point you are updating for.
3 votes
Accepted

Are sentences from the same document independent and identically distributed?

The sentences coming from the same document, author, etc., are unlikely to be independent, that is, the occurrence of a sentence $s_i$ in a certain document $d$ is likely correlated with the ...
  • 34.9k
2 votes

How do I handle negative rewards in policy gradients with the cross-entropy loss function?

The cross-entropy loss will always be positive because the probability is in the range $[0, 1]$, so $-ln(p)$ will always be positive.
2 votes

How to formalize learning in terms of information theory?

In short: It is easy to quantify information, but it is not easy to quantify its usefulness I'm not sure how exactly you are looking to formalise your experiment, but it might be helpful to consider ...
2 votes
Accepted

How does the implementation of the VAE's objective function equate to ELBO?

I don't want to think about the correctness of your supposed ELBO equation now. Nevertheless, it's true that the ELBO can be rewritten in different ways (e.g. if you expand the KL divergence below, by ...
  • 34.9k
2 votes

Why do non-linear activation functions that produce values larger than 1 or smaller than 0 work?

Why wouldn't they work? Each neuron's output is equal to a function over the sum of all its weights multiplied by their corresponding neurons. If that function is the Sigmoid function, then the output ...
2 votes
Accepted

Why is the cross-entropy a cost function?

Optimizing the cross-entropy is equivalent to optimizing the log-likelihood of the parameters given the data, $\ell(\theta)$, which is what we want, i.e. find the parameters that most likely generated ...
  • 34.9k
1 vote

What loss function will be correlated with classification metrics?

Different metrics measure different quantities, so there is no reason to expect different metrics to move together unless one is a function of the other (such as MSE and RMSE). Further, metrics like ...
  • 448
1 vote
Accepted

Trouble writing the backpropagation algorithm in python through crossentropy and softmax

I found the bug on my code, now everything works just fine, so I am fairly sure that the derivation of the formulas is on point. Optimisation wise, clearly using the short formula on the end of the ...
1 vote

In logistic regression, why is the binary cross-entropy loss function convex?

I'm unable to comment on previous answers because I'm new to ai.stackexchange and don't have enough clout points. So I'm writing my comment as an answer instead. Unless I'm missing something, I ...
1 vote

In logistic regression, why is the binary cross-entropy loss function convex?

If you find the Hessian matrix (the matrix of second order derivatives) for the binary cross entropy loss function, you'll see that it is positive semidefinite for any possible value of the parameters....
1 vote
Accepted

What error should I use for RNN?

To provide a good answer would fill several pages. To keep it very simple try many different loss functions on your model. Your goal is to have the highest performance based on some desired ...
1 vote

How to compute the gradient of the cross-entropy loss function with respect to the parameters with softmax activation function?

Alright. Consider an ordinary neural network, so, in the last layer, we have, $z^{[L]} = W^{[L]} a^{[L-1]} + b^{[L]}$, where $a^{[L]} = \sigma(z^{[L]})$, where $\sigma$ is the softmax activation: $$ \...
  • 11
1 vote
Accepted

Have I understood the loss function from the original U-Net paper correctly?

Yes, $E$ is the cross-entropy function and a direct generalization of the binary case. For the binary case, probability to belong to the class $1$ is given by a sigmoid function $\sigma(x)$ of the ...
1 vote

Where is the mistake in my derivation of the GAN loss function?

$\textbf{Remark.}$ I'd leave this as a comment if I could. Regarding notation (which I believe may be the cause of your issue here), the loss function is better written as \begin{align*} \operatorname{...
1 vote

What is the advantage of using cross entropy loss & softmax?

Short answer: larger gradients That is not the derivative of the softmax function. $t - o$ is the combined derivative of the softmax function and cross entropy loss. Cross entropy loss is used to ...
  • 470
1 vote

How do you manage negative rewards in policy gradients?

You don't need to manage negative rewards separately, if you implemented the algorithm correctly it will work regardless if the rewards are negative or not. You seem to be using rewards for the loss ...
  • 2,236
1 vote
Accepted

How should I penalize the model proportionally to the error?

I want to make it so that if the correct label is 3, then it will penalize the model less heavily if it classifies a 4 than a 7 because 4 is closer numerically to 3 than 7 is. How do I do this? ...
  • 24.6k
1 vote

Why does the binary cross-entropy work better than categorical cross-entropy in a multi-class single label problem?

https://stats.stackexchange.com/questions/260505/machine-learning-should-i-use-a-categorical-cross-entropy-or-binary-cross-entro Is relevant. based on my reading when you have a NN and do Binary ...
1 vote

Why do non-linear activation functions that produce values larger than 1 or smaller than 0 work?

Christopher Olah's blog post describes it better that I ever could. Basically, most data we come across can't be separated with a single line, but with some kind of curve. Non-linearities allow us to ...
  • 316

Only top scored, non community-wiki answers of a minimum length are eligible