Questions tagged [cross-entropy]

For questions related to the concept of cross-entropy in the context of artificial intelligence. For example, when the cross-entropy is used as a loss function to train a neural network.

10 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
3 votes
0 answers
331 views

How do I implement the cross-entropy-method for a RL environment with a continuous action space?

I found many tutorials and posts on how to solve RL environments with discrete action spaces using the cross entropy method (e.g., in this blog post for the OpenAI Gym frozen lake environment). ...
Philipp's user avatar
  • 143
3 votes
0 answers
61 views

Is maximum likelihood estimation meaningless for a dataset of only outliers?

From my understanding, maximum likelihood estimation chooses the set of parameters for the estimator that maximizes likelihood with the ground truth distribution. I always interpreted it as the ...
ashenoy's user avatar
  • 1,409
2 votes
1 answer
1k views

How are weights for weighted x-entropy loss on imbalanced data calculated?

I am trying to build a classifier which should be trained with the cross entropy loss. The training data is highly class-imbalanced. To tackle this, I've gone through the advice of the tensorflow docs ...
jmatin's user avatar
  • 21
2 votes
0 answers
205 views

Can the cross-entropy loss be used for a NLP task with LSTM?

I am trying to build an LSTM model to generate Shakspeare-like poems. I have training set $\{s_1,s_2, \dots,s_m\}$, which are sentences of Shakespeare poems, and each sentence contains words $\{w_1,...
Leey's user avatar
  • 43
0 votes
0 answers
33 views

Reinforcement learning - calculating policy gradient using cross entropy loss

I am writing a program that uses reinforcement learning and the policy gradient method to play Pong. It basically extends Andrej Karpathy's version (https://gist.github.com/karpathy/...
Blato's user avatar
  • 1
0 votes
0 answers
21 views

How (or can) you formulate the Fisher information matrix in terms of a loss function, specifically cross-entropy loss?

I recently saw the following formulation of the Fisher information matrix in a paper on Transformer pruning: $$ \mathcal{I} := \frac{1}{|D|} \sum_{(x,y) \in D} \left( \frac{\partial \mathcal{L}(x,y;1)}...
premed's user avatar
  • 1
0 votes
0 answers
169 views

Do LLM or Machine Learning models with a large number of classes employ standard cross entropy?

In language modeling, the next token is predicted from the entire vocabulary during inferenc. However, when training models with large vocabularies or many different classes, standard cross entropy ...
Tommaso Bendinelli's user avatar
0 votes
0 answers
242 views

How to optimize ELBO(VAE's loss function)?

Suppose we've got the following formula: $\log p(x;\theta)=\mathbb{E}_{q(z|x;\phi)}[\log p(x,z;\theta)-\log q(z|x;\phi)]+KL(q(z|x;\phi)||p(z|x;\theta))\\ \geq \mathbb{E}_{q(z|x;\phi)}[\log p(x,z;\...
Garfield's user avatar
0 votes
0 answers
2k views

Focal Loss vs Weighted Cross Entropy Loss

Weighted Focal Loss is defined like so $FL(p_t) = -\alpha_t log(p_t) (1-p_t)^\gamma $ Whereas weighted Cross Entropy Loss is defined like so $CE(p_t) = -\alpha_t log(p_t)$ Some blog posts try to ...
Gulzar's user avatar
  • 759
0 votes
0 answers
37 views

Loss function for better class separability in multi class classification

So I am trying to enforce better separability in my deep learning model and was wondering what I can use besides cross entropy loss to do that? Could maybe using logarithm with different basis in ...
GreatDuke's user avatar
  • 101