Questions tagged [loss]
For questions related to the concept of loss (or cost) in machine learning or other AI sub-fields.
78
questions
0
votes
0
answers
39
views
Why does my loss function fluctuate so much?
I have a loss function that I'm trying to maximise using a neural network.
While it does appear to increase and plateau over the training, it does so in a very "noisy" manner, spiking up and ...
- 73
1
vote
1
answer
55
views
How to transform a loss function into a score function?
Loss_Function/Maximize_Function/Score_Function, CustomLoss, pytorch. Using Custom Loss for Maximizing Score in PyTorch
I'm using a PyTorch model with an LSTM input layer, a linear hidden layer, and 3 ...
- 11
0
votes
1
answer
26
views
What are the differences between loss surfaces that "derive"from different observations?
If I understand right that each observation whithin a dataset, creates a different loss surface where we want to find the global minimum.
How different those surfaces one from another?
Would it be ...
- 181
0
votes
1
answer
61
views
Simple reinforcement learning with human feedback to generate "pleasurable" visual output
I'm trying to build a simple reinforcement learning model that will output a set of parameters that will be passed to a GLSL shader. The human user will rate this visual output, for example "good&...
0
votes
1
answer
69
views
Loss function of logistic Regression Geometric
In linear Regression, I train the Model so the Graph runs best through the Data Points, so the geometric distance between f(x) and y^i is minimized. Now is it correct that in logistic Regression I do ...
- 1
0
votes
0
answers
18
views
Loss is negative- DQN with BCE Loss function
I am writing a code with DQN, using BCE as a loss function for the classification of a sequential time series. But while training, the loss value goes in negative. Also, accuracy and binary accuracy ...
1
vote
1
answer
66
views
Fluctuations in loss during in epoch evaluation of GRU
I am training a one-layer unidirectional vanilla GRU on a next item prediction task with regard to the last 10 interacted items. In my original experiment, where I trained on approx. 5.5M samples and ...
- 33
3
votes
1
answer
322
views
Has anyone tried to train a GPT model predicting the next N tokens instead of the next one token?
I have been thinking about how learning via text works on humans: we read words, and often we need to read ahead a few words to understand more clearly the ideas that we read before. Most of the time, ...
- 33
1
vote
0
answers
60
views
Periodical fluctuations in loss curves
I am training a neural network (specifically a GRU based architecture but I think this is not too relevant for the question). My loss curves, especially the training loss but also the validation loss, ...
- 33
1
vote
0
answers
18
views
Why is my loss graph heavily fluctatuing?
I am working on a CNN project on an image dataset. I am applying Early Stopping technique in order to train the model. However, after training the model and obtaining the loss graph, it is heavily ...
- 11
2
votes
2
answers
171
views
Does MSE loss function work in NN training for predicting values between 0-1?
In a NN regression problem, considering that MSE is squaring the error and the error is between 0 and 1 would it be pointless to use MSE as our loss function during model training?
For example:
...
0
votes
0
answers
139
views
When to stop training object detection network, while taking into the acount 3 factors: loss ,validation loss, mAP
as far as I know, and as can be seen here
enter link description here
it is quite clear that it is better to stop at the "turning point" where the validation loss starts growing.
What I do ...
- 181
0
votes
0
answers
42
views
How does the cross entropy loss function interact with the final layer of a neural network?
I am having trouble understanding how the result of categorical cross entropy loss can be used to calculate the gradient for all of the weights.
The output of cross entropy function is the sum of all ...
- 1
2
votes
0
answers
295
views
GAN : Why does a perfect discriminator mean no gradient for the generator?
In the training of a Generative Adversarial Networks (GAN) system, a perfect discriminator (D) is one which outputs 1 ("true image") for all images of the training dataset and 0 ("false ...
- 221
0
votes
1
answer
220
views
Multilabel text classification with highly imbalanced training data
I'm trying to train a multilabel text classification model using BERT. Each piece of text can belong to 0 or more of a total of 485 classes. My model consists of a dropout layer and a linear layer ...
0
votes
0
answers
42
views
Should the number of training iterations of an instance segmentation model depend on the number of instances in the training dataset?
I need to train instance segmentation models on several different datasets. The datasets vary widely in how many instances each image contains.
For example:
...
- 1
0
votes
0
answers
16
views
Loss function when training on binary_accuracy metric
When training a resnet model with metrics set to binary accuracy does the loss function need to be a binary loss function? Now I am using a custom asymmetric loss function (as my data is very ...
- 1
0
votes
0
answers
8
views
Is a hybrid loss based on hidden (embedding) layer and final classification useful?
Say I have a dataset for classification purposes that I will use to train a CNN. Assume the dataset has a lot of distinguishable details that are not necessarily related to the target labels. Would it ...
- 249
0
votes
1
answer
63
views
Training and validation loss are almost the same (perfect fit?)
I am developing an ANN from scratch which classifies MNIST digits.
These are the curves I get using only one hidden layer composed of 100 neurons activated by ...
- 147
2
votes
2
answers
2k
views
Val loss doesn’t decrease after a certain number of epochs
I’m working on a classification problem (500 classes). My NN has 3 fully connected layers, followed by an LSTM layer. I use nn.CrossEntropyLoss() as my loss ...
- 65
0
votes
1
answer
102
views
Why is `SigmoidBinaryCrossEntropyLoss` in `DJL` implemented this way?
SigmoidBinaryCrossEntropyLoss implementation in DJL accepts two kinds of outputs from NNs:
where sigmoid activation has already been applied.
where raw NN output ...
- 1
0
votes
1
answer
159
views
What does the adversarial loss in a GAN represent?
I'm working on Pix2Pix an image-to-image translation GAN, and I noticed that there is an adversarial loss implemented using BCE, and a L1 loss implemented using MAE. I know L1 loss represents the ...
- 11
0
votes
1
answer
62
views
Training loss decreases very fast after few epochs
I am implementing an ANN whose training loss is in Figure:
As you can see training loss decreases very fast and it is approximately 3.2 at epochs 2, 3, ..., 8, ... 10, and so on. (batch learning)
The ...
- 147
1
vote
0
answers
60
views
Training a neural network simultaneously with two different loss functions rather than considering the weighted sum
This is a follow up on the already asked question: Is the neural network 100% accurate on training data if epoch loss is minimized to 0?
I want to train a neural network that works as an approximator ...
- 111
0
votes
1
answer
73
views
How to use Categorical Cross Entropy for Multi-Label Classification?
Say my target with classes A, B, C, D, E is [0, 1, 1, 0, 0].
And my output layer is of B x N where N is the number of classes.
...
- 31
1
vote
1
answer
62
views
What loss function will be correlated with classification metrics?
Recently I developed a custom training algorithm for deep learning models, based on evolutionary algorithms. Details are not important, except that it also uses decreasing regular cross entropy loss ...
- 1,240
1
vote
1
answer
44
views
Do we need to know or verify properties of loss functions / metrics' implementations?
I will start with an example, in order to get to the general question.
I was reading the following paper (https://www.cns.nyu.edu/pub/lcv/wang03-preprint.pdf) about Structural Similarity Index (SSIM), ...
- 185
0
votes
1
answer
68
views
Can MSE be used for NN categorical classification problems
I currently have a neural network that can manage to perform polynomial (single output) regression problems. I now want to upscale to classification problems (eg: image recognition). Can I do this ...
- 13
0
votes
1
answer
98
views
Non-Convex loss-surface although quadratic loss function
there is one problem which bugs me quite a long time, it is the non-convex loss shape (multiple minima, e.g. shown here) of neural networks which use a quadratic loss function.
Question: Why is a “...
- 3
1
vote
0
answers
69
views
How to detect peak locations via Neural Networks?
As part of my masters thesis, I'm developing generative models for ECGs.
Right now, I have a Denoising Diffusion Implicit model (DDIM), that transforms random noise into a valid ECG (2s long, or 1024 ...
- 11
1
vote
1
answer
41
views
How do I interpret this loss function?
In this AI note from https://deeplearning.ai, the loss function below is used for a regression problem. However, I don't know how to interpret this loss function.
First, does the author take the ...
- 113
1
vote
0
answers
227
views
Categorical loss function for variable number of labels
I have a model for binary classification. The target variable has the different number of labels (instances) in each sample. For example, a batch of size 2 with 2 and 3 instances and correspondingly ...
- 111
0
votes
1
answer
27
views
"Tweaking" the cost function to penalize rarer cases more severely
I have a very unbalanced data set that I am running a CNN on for regression. Most of the values are 0, while it is possible for the values to range from 0 to 32.
Is it possible to "tweak" ...
- 125
0
votes
0
answers
74
views
uniform gap between training and validation metrics
I am training a neural network (Deep and cross network) for a multi-label classification task (~700 labels). I have around 2.5 million samples, splitted 8/1/1 for train/test/validation. I am seeing a ...
- 121
2
votes
1
answer
172
views
Why do we subtract logsumexp from the outputs of this neural network?
I'm trying to understand this tutorial for Jax.
Here's an excerpt. It's for a neural net that is designed to classify MNIST images:
...
- 151
2
votes
1
answer
651
views
What is being optimized with WGAN loss? Is the generator maximizing or minimizing the critic value?
I am kind of new to the field of GANs and decided to develop a WGAN. All of the information online seems to be kind of contradicting itself. The more I read, the more I become confused, so I'm hoping ...
1
vote
1
answer
74
views
Visualizing the loss landscape in deep NN to compare optimization methods
I'm comparing 2 optimization algorithms for deep neural nets through visualizing the loss landscape. The visualization method is described here.
Besides the qualitative observation that how trajectory ...
- 151
0
votes
2
answers
58
views
What does IOU3 mean in this context?
I was reading a paper and this paragraph said that:
The ground truth score is calculated based on the intersectionover-
union (IoU) of the perturbed image and the ground truth one. Since we
would ...
- 3
1
vote
2
answers
257
views
What is the domain of the discriminator of a GAN?
I've read that the discriminator $D$ validates an image $D(x)$, where $x$ is either a real image or a fake one created by the generator, i.e. $ D(G(x))$.
What does the function of the discriminator ...
- 19
1
vote
1
answer
751
views
PPO: policy loss becomes nan [closed]
I'm implement PPO for a very specific problem, and it seems to be working somewhat, but after a few epochs, I always get something like this:
...
2
votes
2
answers
585
views
Why does triplet loss allow to learn a ranking whereas contrastive loss only allows to learn similarity?
I am looking at this lecture, which states (link to exact time):
What the triplet loss allows us in contrast to the contrastive loss is
that we can learn a ranking. So it's not only about similarity, ...
- 729
0
votes
0
answers
33
views
What is the correct formula for the loss function?
I have used the Delayed sin echo prediction with Tensorflow that predicts the sin wave. However, I'm not sure of the correct formula for the loss function. The problem is that I feed the training ...
- 52
1
vote
1
answer
158
views
How contrastive loss work intuitively in siamese network
I am having issue in getting clear concept of contrastive loss used in siamese network.
Here is pytorch formula
...
- 123
0
votes
1
answer
170
views
Does this modified version of the triplet loss function introduced with SBERT that uses the cosine similarity make sense?
I am working on a modified version of the triplet loss function introduced with SBERT, where instead of the Euclidean distance we use the cosine similarity. The formula to minimize is ...
- 105
1
vote
0
answers
346
views
How to compute circle loss?
I have read the paper about circle loss for neural networks. I may have missed something, but I didn't find the way to compute the positive similarity and the negative similarity in the case of circle ...
- 111
1
vote
1
answer
773
views
Does Value Loss in Actor Critic not decrease at all?
I am coding a problem with the Actor-Critic Method. The final loss is a summation of PolicyLoss and ValueLoss. The calculation of the PolicyLoss for each step is given at Equation Number 5 of https://...
0
votes
0
answers
2k
views
Focal Loss vs Weighted Cross Entropy Loss
Weighted Focal Loss is defined like so
$FL(p_t) = -\alpha_t log(p_t) (1-p_t)^\gamma $
Whereas weighted Cross Entropy Loss is defined like so
$CE(p_t) = -\alpha_t log(p_t)$
Some blog posts try to ...
- 729
0
votes
1
answer
1k
views
Is learning rate the only reason for training loss oscillation after few epochs?
Consider the following loss curve
The x-axis is the no. of epochs and the y-axis is the loss function.
You can observe that loss is decreasing drastically for the first few epochs and then starts ...
- 3,581
1
vote
0
answers
148
views
How to compute the loss for a sequence labeling task without the Softmax distribution?
For a sequence labeling task (NER), we compute the loss by passing the softmax distribution of the classes (e.g. vocabulary) with the gold label to the loss function (...
- 123
1
vote
1
answer
27
views
Writing a loss function for "how far can this output be pushed"
I'm trying to train a function for a industrial-process-control-like system. This is my first attempt at a custom training, so feel free to point out any invalid assumptions.
I've got one input and ...
- 111