16
votes
Accepted
Why has the cross-entropy become the classification standard loss function and not Kullback-Leibler divergence?
When it comes to a classification problem in machine learning, the cross-entropy and the KL divergence are equal.
As already stated in the question, the general formula is this:
$$H(p, q) = H(p) + D_{...
- 1,947
7
votes
How do I handle negative rewards in policy gradients with the cross-entropy loss function?
It depends on your loss function, but you probably need to tweak it.
If you are using an update rule like loss = -log(probabilities) * reward, then your loss is ...
- 171
5
votes
Accepted
Why are there two versions of softmax cross entropy? Which one to use in what situation?
It's the same thing, first version is the special case of the more general one. In the first case you only have two classes, it's binary cross-entropy, and they also included iteration over batch of ...
- 2,316
4
votes
Accepted
Where is the mistake in my derivation of the GAN loss function?
I guess the issue is you lost track of where the samples came from and since you requested a math explanation I'll try to go step by step using my notation and without checking other material to avoid ...
- 799
4
votes
Accepted
How does the implementation of the VAE's objective function equate to ELBO?
I don't want to think about the correctness of your supposed ELBO equation now. Nevertheless, it's true that the ELBO can be rewritten in different ways (e.g. if you expand the KL divergence below, by ...
- 39.1k
3
votes
In logistic regression, why is the binary cross-entropy loss function convex?
The $L_{CE}$ that you provided is binary cross-entropy, the factor $y$ and $(1-y)$ is because $y$ is binary $({0,1})$, careful with the name next time. The cross-entropy loss should have form:
$$L_{CE}...
- 892
3
votes
How do I handle negative rewards in policy gradients with the cross-entropy loss function?
The cross-entropy loss will always be positive because the probability is in the range $[0, 1]$, so $-ln(p)$ will always be positive.
- 31
3
votes
How to formalize learning in terms of information theory?
In short: It is easy to quantify information, but it is not easy to quantify its usefulness
I'm not sure how exactly you are looking to formalise your experiment, but it might be helpful to consider ...
3
votes
Accepted
Are sentences from the same document independent and identically distributed?
The sentences coming from the same document, author, etc., are unlikely to be independent, that is, the occurrence of a sentence $s_i$ in a certain document $d$ is likely correlated with the ...
- 39.1k
3
votes
Accepted
Which loss function should I use in REINFORCE, and what are the labels?
The loss function you are looking for is cross entropy loss. The 'label' that you use is the action you took at the time point you are updating for.
- 4,615
2
votes
Why do non-linear activation functions that produce values larger than 1 or smaller than 0 work?
Why wouldn't they work?
Each neuron's output is equal to a function over the sum of all its weights multiplied by their corresponding neurons. If that function is the Sigmoid function, then the output ...
- 906
2
votes
Accepted
Why is the cross-entropy a cost function?
Optimizing the cross-entropy is equivalent to optimizing the log-likelihood of the parameters given the data, $\ell(\theta)$, which is what we want, i.e. find the parameters that most likely generated ...
- 39.1k
2
votes
What loss function will be correlated with classification metrics?
Different metrics measure different quantities, so there is no reason to expect different metrics to move together unless one is a function of the other (such as MSE and RMSE).
Further, metrics like ...
- 538
2
votes
why cross entropy loss has to be multiplied by a batch size during an evaluation in transformer model?
That is because the default reduction for the CEP loss calculation is mean. Hence to find the true average across all batches, you first multiply by the batch size and then divide by total number of ...
- 21
1
vote
Backpropagation with CrossEntropy and Softmax, HOW?
The derivation of the softmax function is a bit tricky because except the other common activation functions(sigmoid, relu...) all the Zs values impact each other. It is because when you are ...
- 25
1
vote
Backpropagation with CrossEntropy and Softmax, HOW?
No it's not the same. First the derivative of the cost function is taken with respect to the weight and not the input. This is usually done using the chain rule of calculus. To calculate this using ...
- 11
1
vote
Accepted
What is the correct loss function for binary classification: Cross entropy or Binary cross entropy?
Binary cross-entropy loss is a specific case for cross-entropy loss. Theoretically, one can also use the normal cross-entropy loss for binary classification. Binary cross-entropy is probably ...
- 1,898
1
vote
Accepted
Trouble writing the backpropagation algorithm in python through crossentropy and softmax
I found the bug on my code, now everything works just fine, so I am fairly sure that the derivation of the formulas is on point. Optimisation wise, clearly using the short formula on the end of the ...
- 121
1
vote
In logistic regression, why is the binary cross-entropy loss function convex?
I'm unable to comment on previous answers because I'm new to ai.stackexchange and don't have enough clout points. So I'm writing my comment as an answer instead.
Unless I'm missing something, I ...
1
vote
In logistic regression, why is the binary cross-entropy loss function convex?
If you find the Hessian matrix (the matrix of second order derivatives) for the binary cross entropy loss function, you'll see that it is positive semidefinite for any possible value of the parameters....
- 111
1
vote
Accepted
What error should I use for RNN?
To provide a good answer would fill several pages. To keep it very simple try many different loss functions on your model. Your goal is to have the highest performance based on some desired ...
- 1,853
1
vote
How to compute the gradient of the cross-entropy loss function with respect to the parameters with softmax activation function?
Alright. Consider an ordinary neural network, so, in the last layer, we have, $z^{[L]} = W^{[L]} a^{[L-1]} + b^{[L]}$, where $a^{[L]} = \sigma(z^{[L]})$, where $\sigma$ is the softmax activation:
$$
\...
- 11
1
vote
Accepted
Have I understood the loss function from the original U-Net paper correctly?
Yes, $E$ is the cross-entropy function and a direct generalization of the binary case.
For the binary case, probability to belong to the class $1$ is given by a sigmoid function $\sigma(x)$ of the ...
- 2,494
1
vote
Where is the mistake in my derivation of the GAN loss function?
$\textbf{Remark.}$ I'd leave this as a comment if I could.
Regarding notation (which I believe may be the cause of your issue here), the loss function is better written as
\begin{align*}
\operatorname{...
- 11
1
vote
What is the advantage of using cross entropy loss & softmax?
Short answer: larger gradients
That is not the derivative of the softmax function. $t - o$ is the combined derivative of the softmax function and cross entropy loss. Cross entropy loss is used to ...
- 560
1
vote
How do you manage negative rewards in policy gradients?
You don't need to manage negative rewards separately, if you implemented the algorithm correctly it will work regardless if the rewards are negative or not. You seem to be using rewards for the loss ...
- 2,316
1
vote
Accepted
How should I penalize the model proportionally to the error?
I want to make it so that if the correct label is 3, then it will penalize the model less heavily if it classifies a 4 than a 7 because 4 is closer numerically to 3 than 7 is. How do I do this?
...
- 29k
1
vote
Accepted
Why does PyTorch use a different formula for the cross-entropy?
When you one-hot-encode your labels with $p_i \in \{0,1\}$ you get $p_i = 0$ iff $i$ is not correct and, equivalently, $p_i =1$ iff $i$ is correct.
Hence, $p_i \log(q_i) = 0 \log(q_i) = 0 $ for all ...
- 304
1
vote
Why does the binary cross-entropy work better than categorical cross-entropy in a multi-class single label problem?
https://stats.stackexchange.com/questions/260505/machine-learning-should-i-use-a-categorical-cross-entropy-or-binary-cross-entro
Is relevant.
based on my reading when you have a NN and do Binary ...
- 545
Only top scored, non community-wiki answers of a minimum length are eligible