Share Your Experience: Take the 2024 Developer Survey

# Questions tagged [cross-entropy]

For questions related to the concept of cross-entropy in the context of artificial intelligence. For example, when the cross-entropy is used as a loss function to train a neural network.

37 questions
Filter by
Sorted by
Tagged with
9 views

### Incredibly High CrossEntropyLoss in Sequence-to-Sequence Generation

I'm trying to do SMILES chemical representation prediction from a large dataset (Around 5M Samples) to teach it do predict another downstream task. The model's part responsible for generating the data ...
• 101
43 views

### Reinforcement learning - calculating policy gradient using cross entropy loss

I am writing a program that uses reinforcement learning and the policy gradient method to play Pong. It basically extends Andrej Karpathy's version (https://gist.github.com/karpathy/...
87 views

1 vote
80 views

### What is the lowest possible loss for a language model?

Example: Suppose a character-level language model (three input letters to predict the next one), trained on a dataset which contains three instances of the sequence ...
1 vote
172 views

### Backpropagation with CrossEntropy and Softmax, HOW?

Let Zs be the input of the output layer (for example, Z1 is the input of the first neuron in the output layer), Os be the output of the output layer (which are actually the results of applying the ...
• 11
389 views

### What is the correct loss function for binary classification: Cross entropy or Binary cross entropy?

Let's say I have a binary classification problem and I want to solve it by means of FC neural net. So which approach will be correct: 1) define the last layer of NN like this ...
• 23
286 views

• 113
2k views

### What is the advantage of using cross entropy loss & softmax?

I am trying to do the standard MNIST dataset image recognition test with a standard feed forward NN, but my network failed pretty badly. Now I have debugged it quite a lot and found & fixed some ...
• 435
39 views

### Loss function for better class separability in multi class classification

So I am trying to enforce better separability in my deep learning model and was wondering what I can use besides cross entropy loss to do that? Could maybe using logarithm with different basis in ...
• 101
1k views

### Which loss function should I use in REINFORCE, and what are the labels?

I understand that this is the update for the parameters of a policy in REINFORCE: $$\Delta \theta_{t}=\alpha \nabla_{\theta} \log \pi_{\theta}\left(a_{t} \mid s_{t}\right) v_{t},$$ where $v_t$ is ...
• 590
1k views

### How do you manage negative rewards in policy gradients?

This old question has no definitive answer yet, that's why I am asking it here again. I also asked this same question here. If I'm doing policy gradient in Keras, using a loss of the form: ...
• 121
1k views

### How are weights for weighted x-entropy loss on imbalanced data calculated?

I am trying to build a classifier which should be trained with the cross entropy loss. The training data is highly class-imbalanced. To tackle this, I've gone through the advice of the tensorflow docs ...
• 21
63 views

### Is maximum likelihood estimation meaningless for a dataset of only outliers?

From my understanding, maximum likelihood estimation chooses the set of parameters for the estimator that maximizes likelihood with the ground truth distribution. I always interpreted it as the ...
• 1,419
211 views

• 43
121 views

### How should I penalize the model proportionally to the error?

I am making an MNIST classifier. I am using categorical cross-entropy as my loss function. I want to make it so that if the correct label is 3, then it will penalize the model less heavily if it ...
1 vote
143 views

### Why does PyTorch use a different formula for the cross-entropy?

In my understanding, the formula to calculate the cross-entropy is $$H(p,q) = - \sum p_i \log(q_i)$$ But in PyTorch nn.CrossEntropyLoss is calculated using this ...
• 2,819
125 views

### How to formalize learning in terms of information theory?

Consider the following game on a MNIST dataset: There are 60000 images. You can pick any 1000 images and train your Neural Network without access to the rest of images. Your final result is ...
• 141
570 views

### Why does the binary cross-entropy work better than categorical cross-entropy in a multi-class single label problem?

I was just doing a simple NN example with the fashion MNIST dataset, where I was getting 97% accuracy, when I noticed that I was using Binary cross-entropy instead of categorical cross-entropy by ...
8k views

### How is division by zero avoided when implementing back-propagation for a neural network with sigmoid at the output neuron?

I am building a neural network for which I am using the sigmoid function as the activation function for the single output neuron at the end. Since the sigmoid function is known to take any number and ...
• 73
515 views

### Why do non-linear activation functions that produce values larger than 1 or smaller than 0 work?

Why do non-linear activation functions that produce values larger than 1 or smaller than 0 work? My understanding is that neurons can only produce values between 0 and 1, and that this assumption can ...
5k views

### Why has the cross-entropy become the classification standard loss function and not Kullback-Leibler divergence?

The cross-entropy is identical to the KL divergence plus the entropy of the target distribution. The KL divergence equals zero when the two distributions are the same, which seems more intuitive to me ...
• 301