Skip to main content
Share Your Experience: Take the 2024 Developer Survey

Questions tagged [cross-entropy]

For questions related to the concept of cross-entropy in the context of artificial intelligence. For example, when the cross-entropy is used as a loss function to train a neural network.

Filter by
Sorted by
Tagged with
0 votes
0 answers
9 views

Incredibly High CrossEntropyLoss in Sequence-to-Sequence Generation

I'm trying to do SMILES chemical representation prediction from a large dataset (Around 5M Samples) to teach it do predict another downstream task. The model's part responsible for generating the data ...
Vivek Joshy's user avatar
0 votes
0 answers
43 views

Reinforcement learning - calculating policy gradient using cross entropy loss

I am writing a program that uses reinforcement learning and the policy gradient method to play Pong. It basically extends Andrej Karpathy's version (https://gist.github.com/karpathy/...
Blato's user avatar
  • 1
2 votes
2 answers
87 views

Why is the cross entropy formula different from the binary cross entropy?

why cross entropy formulation is different with binary cross entropy? cross entorpy loss $$ H_p(q) = -\sum_{q_i}^C [q_i \log(p_i)] $$ binary cross entorpy $$ -\sum [k_i \log(p_i)+(1-k_i) \log(1-p_i)] ...
Shin Joong Hyun's user avatar
0 votes
0 answers
24 views

How (or can) you formulate the Fisher information matrix in terms of a loss function, specifically cross-entropy loss?

I recently saw the following formulation of the Fisher information matrix in a paper on Transformer pruning: $$ \mathcal{I} := \frac{1}{|D|} \sum_{(x,y) \in D} \left( \frac{\partial \mathcal{L}(x,y;1)}...
premed's user avatar
  • 1
1 vote
1 answer
80 views

What is the lowest possible loss for a language model?

Example: Suppose a character-level language model (three input letters to predict the next one), trained on a dataset which contains three instances of the sequence ...
ViniciusArruda's user avatar
1 vote
2 answers
172 views

Backpropagation with CrossEntropy and Softmax, HOW?

Let Zs be the input of the output layer (for example, Z1 is the input of the first neuron in the output layer), Os be the output of the output layer (which are actually the results of applying the ...
qazaq's user avatar
  • 11
0 votes
1 answer
389 views

What is the correct loss function for binary classification: Cross entropy or Binary cross entropy?

Let's say I have a binary classification problem and I want to solve it by means of FC neural net. So which approach will be correct: 1) define the last layer of NN like this ...
dmasny's user avatar
  • 23
0 votes
0 answers
286 views

How to optimize ELBO(VAE's loss function)?

Suppose we've got the following formula: $\log p(x;\theta)=\mathbb{E}_{q(z|x;\phi)}[\log p(x,z;\theta)-\log q(z|x;\phi)]+KL(q(z|x;\phi)||p(z|x;\theta))\\ \geq \mathbb{E}_{q(z|x;\phi)}[\log p(x,z;\...
Garfield's user avatar
1 vote
1 answer
68 views

What loss function will be correlated with classification metrics?

Recently I developed a custom training algorithm for deep learning models, based on evolutionary algorithms. Details are not important, except that it also uses decreasing regular cross entropy loss ...
GKozinski's user avatar
  • 1,270
0 votes
1 answer
1k views

why cross entropy loss has to be multiplied by a batch size during an evaluation in transformer model?

I am trying to look through a code of the transformer model from Pytorch. However, I do not understand why batch size needs to multiply with cross-entropy loss given that loss is calculated based on ...
BOB's user avatar
  • 1
1 vote
1 answer
186 views

Trouble writing the backpropagation algorithm in python through crossentropy and softmax

so I am writing my own neural network library for a class project and I got everything working for a simple 2-class test using the distance (L2) cost function. I wanted to get a similar result using ...
user605734 MBS's user avatar
1 vote
0 answers
2k views

Focal Loss vs Weighted Cross Entropy Loss

Weighted Focal Loss is defined like so $FL(p_t) = -\alpha_t log(p_t) (1-p_t)^\gamma $ Whereas weighted Cross Entropy Loss is defined like so $CE(p_t) = -\alpha_t log(p_t)$ Some blog posts try to ...
Gulzar's user avatar
  • 769
-1 votes
1 answer
219 views

Why is the cross-entropy a cost function?

The question looks foolish, but I think cross-entropy is somewhat weird as a cost function. As a cost function for linear regression, the mean square error $ \sum_{i=1}^{n} (y_i - (ax_i+b)) ^2$ seems ...
JAEMTO's user avatar
  • 125
5 votes
3 answers
5k views

In logistic regression, why is the binary cross-entropy loss function convex?

I am studying logistic regression for binary classification. The loss function used is cross-entropy. For a given input $x$, if our model outputs $\hat{y}$ instead of $y$, the loss is given by $$\text{...
hanugm's user avatar
  • 3,890
3 votes
0 answers
342 views

How do I implement the cross-entropy-method for a RL environment with a continuous action space?

I found many tutorials and posts on how to solve RL environments with discrete action spaces using the cross entropy method (e.g., in this blog post for the OpenAI Gym frozen lake environment). ...
Philipp's user avatar
  • 143
0 votes
1 answer
201 views

What error should I use for RNN?

I'm relatively new to machine learning, and I don't know what error I should use for an RNN. I want to use a simple Elman RNN to predict the cases of Covid-19 there will be in a hospital for the next ...
mariogarcia's user avatar
1 vote
1 answer
662 views

How to compute the gradient of the cross-entropy loss function with respect to the parameters with softmax activation function?

I've seen plenty of examples of people doing Sigmoid + MSE backpropagation implementations, yet I do not seem to understand how to implement backpropagation as stated in the title in the case of multi-...
Ilknur Mustafa's user avatar
1 vote
1 answer
676 views

Have I understood the loss function from the original U-Net paper correctly?

In the original U-Net paper, it is written The energy function is computed by a pixel-wise soft-max over the final feature map combined with the cross entropy loss function. ... $$ E=\sum_{\mathbf{x} ...
Bert Gayus's user avatar
3 votes
2 answers
301 views

Where is the mistake in my derivation of the GAN loss function?

I was pondering on the loss function of GAN, and the following thing turned out \begin{aligned} L(D, G) & = \mathbb{E}_{x \sim p_{r}(x)} [\log D(x)] + \mathbb{E}_{x \sim p_g(x)} [\log(1 - D(x)] \...
Enes's user avatar
  • 324
2 votes
1 answer
2k views

How does the implementation of the VAE's objective function equate to ELBO?

For a lot of VAE implementations I've seen in code, it's not really obvious to me how it equates to ELBO. $$L(X)=H(Q)-H(Q:P(X,Z))=\sum_ZQ(Z)logP(Z,X)-\sum_ZQ(Z)log(Q(Z))$$ The above is the definition ...
user8714896's user avatar
1 vote
1 answer
309 views

Why are there two versions of softmax cross entropy? Which one to use in what situation?

I have seen 2 forms of softmax cross-entropy loss and are confused by the two. Which one is the right one? For example in this Quora answer, there are 2 answers: $L(\mathbf{w})=\frac{1}{N} \sum_{n=1}^...
Herbert's user avatar
  • 113
5 votes
2 answers
2k views

What is the advantage of using cross entropy loss & softmax?

I am trying to do the standard MNIST dataset image recognition test with a standard feed forward NN, but my network failed pretty badly. Now I have debugged it quite a lot and found & fixed some ...
Ben's user avatar
  • 435
0 votes
0 answers
39 views

Loss function for better class separability in multi class classification

So I am trying to enforce better separability in my deep learning model and was wondering what I can use besides cross entropy loss to do that? Could maybe using logarithm with different basis in ...
GreatDuke's user avatar
  • 101
8 votes
1 answer
1k views

Which loss function should I use in REINFORCE, and what are the labels?

I understand that this is the update for the parameters of a policy in REINFORCE: $$ \Delta \theta_{t}=\alpha \nabla_{\theta} \log \pi_{\theta}\left(a_{t} \mid s_{t}\right) v_{t}, $$ where $v_t$ is ...
S2673's user avatar
  • 590
2 votes
1 answer
1k views

How do you manage negative rewards in policy gradients?

This old question has no definitive answer yet, that's why I am asking it here again. I also asked this same question here. If I'm doing policy gradient in Keras, using a loss of the form: ...
Mastiff's user avatar
  • 121
2 votes
1 answer
1k views

How are weights for weighted x-entropy loss on imbalanced data calculated?

I am trying to build a classifier which should be trained with the cross entropy loss. The training data is highly class-imbalanced. To tackle this, I've gone through the advice of the tensorflow docs ...
jmatin's user avatar
  • 21
3 votes
0 answers
63 views

Is maximum likelihood estimation meaningless for a dataset of only outliers?

From my understanding, maximum likelihood estimation chooses the set of parameters for the estimator that maximizes likelihood with the ground truth distribution. I always interpreted it as the ...
ashenoy's user avatar
  • 1,419
2 votes
0 answers
211 views

Can the cross-entropy loss be used for a NLP task with LSTM?

I am trying to build an LSTM model to generate Shakspeare-like poems. I have training set $\{s_1,s_2, \dots,s_m\}$, which are sentences of Shakespeare poems, and each sentence contains words $\{w_1,...
Leey's user avatar
  • 43
1 vote
1 answer
75 views

Are sentences from the same document independent and identically distributed?

I am trying to build an LSTM model to generate Shakspeare-like poems. I have data set $\{s_1, s_2, \dots, s_m \}$, which are sentences of Shakespeare poems, and each sentence contains words $\{w_1, ...
Leey's user avatar
  • 43
2 votes
1 answer
121 views

How should I penalize the model proportionally to the error?

I am making an MNIST classifier. I am using categorical cross-entropy as my loss function. I want to make it so that if the correct label is 3, then it will penalize the model less heavily if it ...
Josh Goldman's user avatar
1 vote
1 answer
143 views

Why does PyTorch use a different formula for the cross-entropy?

In my understanding, the formula to calculate the cross-entropy is $$ H(p,q) = - \sum p_i \log(q_i) $$ But in PyTorch nn.CrossEntropyLoss is calculated using this ...
malioboro's user avatar
  • 2,819
4 votes
1 answer
125 views

How to formalize learning in terms of information theory?

Consider the following game on a MNIST dataset: There are 60000 images. You can pick any 1000 images and train your Neural Network without access to the rest of images. Your final result is ...
Oleg Dats's user avatar
  • 141
4 votes
1 answer
570 views

Why does the binary cross-entropy work better than categorical cross-entropy in a multi-class single label problem?

I was just doing a simple NN example with the fashion MNIST dataset, where I was getting 97% accuracy, when I noticed that I was using Binary cross-entropy instead of categorical cross-entropy by ...
joão correia's user avatar
7 votes
1 answer
8k views

How is division by zero avoided when implementing back-propagation for a neural network with sigmoid at the output neuron?

I am building a neural network for which I am using the sigmoid function as the activation function for the single output neuron at the end. Since the sigmoid function is known to take any number and ...
Dimitry's user avatar
  • 73
2 votes
2 answers
515 views

Why do non-linear activation functions that produce values larger than 1 or smaller than 0 work?

Why do non-linear activation functions that produce values larger than 1 or smaller than 0 work? My understanding is that neurons can only produce values between 0 and 1, and that this assumption can ...
Emil Wormbs's user avatar
19 votes
1 answer
5k views

Why has the cross-entropy become the classification standard loss function and not Kullback-Leibler divergence?

The cross-entropy is identical to the KL divergence plus the entropy of the target distribution. The KL divergence equals zero when the two distributions are the same, which seems more intuitive to me ...
Josh Albert's user avatar
10 votes
2 answers
12k views

How do I handle negative rewards in policy gradients with the cross-entropy loss function?

I am using policy gradients in my reinforcement learning algorithm, and occasionally my environment provides a severe penalty (i.e. negative reward) when a wrong move is made. I'm using a neural ...
jstaker7's user avatar
  • 209