Questions tagged [objective-functions]
For questions related to the concept of loss (or cost) function in the context of machine learning.
253
questions
0
votes
1
answer
23
views
What is the correct loss function for binary classification: Cross entropy or Binary cross entropy?
Let's say I have a binary classification problem and I want to solve it by means of FC neural net. So which approach will be correct: 1) define the last layer of NN like this ...
- 3
0
votes
1
answer
34
views
What's the difference between classification and segmentation in deep learning?
What's the difference between classification and segmentation in deep learning?
In particular, can the classification loss function be used for segmentation problems?
0
votes
0
answers
34
views
How do Deep Momentum Networks work?
Here is a paper about Deep Momentum Networks: https://arxiv.org/pdf/1904.04912.pdf
From what I understand, they are a neural network that's used for stock trading, that uses the Sharpe Ratio as a loss ...
- 15
0
votes
0
answers
22
views
What loss function should I use to penalize shift properly
I'm trying to fit a set of parameters $\mathbf{p} \in \mathbb{R}^P$ to a 1D function $\hat{f}(t)$ (e.g. waveform, time-series) where $t\in\mathbb{R}$ is the time coordinate of the signal $\hat{f}\in\...
- 111
2
votes
2
answers
89
views
Image classification problem with multiple right classes
I have a use case where the model needs to detect fabricdefects. There are 15+ different kinds of defects. In one image there can be multiple defects present. The straight forward solution for this ...
1
vote
1
answer
143
views
Why MSE and MAE yield poor results when used with gradient-based optimization for classification?
Deep learning book chapter 6: In 6.2.1.2 last paragraph:
Unfortunately, mean squared error and mean absolute error often lead to poor results when used with gradient-based optimization. Some output ...
- 13
0
votes
0
answers
21
views
What approach finds an aproximation to a function provided only score?
I want to approximate an expensive function without having the training data of correct input-output pairs, instead having the learning model quarry specific input-output pairs and my supervisor (if ...
0
votes
1
answer
96
views
Why is `SigmoidBinaryCrossEntropyLoss` in `DJL` implemented this way?
SigmoidBinaryCrossEntropyLoss implementation in DJL accepts two kinds of outputs from NNs:
where sigmoid activation has already been applied.
where raw NN output ...
- 1
1
vote
0
answers
13
views
Loss Function for Binary Classification with Multiple Correct Choices
I have a binary classification problem, where there are multiple correct predictions, however, I would consider the prediction to be correct if the highest confidence prediction of a 1 is correct.
I ...
- 11
0
votes
1
answer
26
views
Learning curve converges with huge errors
I am training an auto-encoder over $10^4$ epochs. I get a converging learning curve. However the error at the last stages stays huge $\sim10^{15}$. What does this mean? does it mean that my auto-...
- 101
1
vote
0
answers
41
views
Training a neural network simultaneously with two different loss functions rather than considering the weighted sum
This is a follow up on the already asked question: Is the neural network 100% accurate on training data if epoch loss is minimized to 0?
I want to train a neural network that works as an approximator ...
- 111
0
votes
0
answers
59
views
Left-to-Right vs Encoder-decoder Models
Xu et al. (2022) distinguishes between popular pre-training methods for language modeling: (see Section 2.1 PRETRAINING METHODS)
Left-to-Right:
Auto-regressive, Left-to-right models, predict the ...
- 29
0
votes
0
answers
36
views
Combining loss function while training with another objective function
I would like to train a ReLU neural network minimizing an objective function that looks like this:
$$L(W) + \eta + 1_{S}(\eta,W)$$ where $W$ is the set of weight matrices, $L(W)$ is a custom loss ...
- 111
1
vote
1
answer
36
views
Do we need to know or verify properties of loss functions / metrics' implementations?
I will start with an example, in order to get to the general question.
I was reading the following paper (https://www.cns.nyu.edu/pub/lcv/wang03-preprint.pdf) about Structural Similarity Index (SSIM), ...
- 155
0
votes
0
answers
12
views
Choosing an appropriate loss function for sparse label proportion estimation
I'm working over a task of estimating sparse label proportions, where the target is probability distribution $\textbf{q} \in \Delta^{K-1}$ and $\Delta^{K-1} := \{\textbf{p} \in \mathbb{R}^K \, | \, ...
- 277
0
votes
0
answers
30
views
Why does a quantile regression estimator underestimate the variance when using the quantile huber loss?
I have a question to quantile regression which is related to distributional Reinforcement Learning. Let the quantile loss (QL) be defined as
\begin{align*}
\mathcal{L}^{\tau}_{\text{QR}}(\...
- 55
0
votes
0
answers
12
views
Same 'area under curve' but different parameters?
For 'area under the curve' (AUC) calculations in machine learning, is it the case that if we are mapping the false positives (x-axis) against the true positives (y-axis), that this curve must be ...
0
votes
0
answers
10
views
Tree boosting additive loss
In the XGBoost documentation, they specify that the additive training is done given an objective $obj^{(t)}$ defined as
$obj^{(t)} = \sum\limits_{i=1}^n \ell(y_i, \hat{y}_i^{(t-1)}+f_t(x_i)) + \sum\...
- 341
0
votes
1
answer
40
views
Is the discriminator of a GAN network embedded in VAE?
From what I understand, a Generative Adversarial Network (GAN) is composed of an encoder (generator), some synthetic data (fake data) and a discriminator that will penalize any distinguishable real ...
- 3
3
votes
1
answer
198
views
What loss function should I use if I only care about the accuracy of one class?
CrossEntropyLoss optimizes the overall classification accuracy as
$$ {n_{\text{correct}} \over N} $$
What loss function should I use if I only care about increasing the true positive rate of one class?...
0
votes
0
answers
23
views
Why would the training loss consistently increase over many epochs?
I am getting a very strange learning curve when I try training a neural network which I am not able to explain.
I have never seen a learning curve that looks like this when it's a very simple and ...
- 101
0
votes
0
answers
111
views
Why Pix2Pix loss function does not lead to overfitting?
In the Pix2Pix paper, the loss function is described as
$\ \mathcal{L}_{cGAN}(G,D) + \lambda \mathcal{L}_{L1}(G)$ . Where the L1 loss in the model is the difference of pixels between the generated ...
- 35
0
votes
2
answers
57
views
How to define a loss function for multi-label problem?
I have voice recordings which are labelled by not only a single label but multiple labels. Each voice recording corresponds to one of class labels within a set. In other words, the training instance ...
- 103
0
votes
0
answers
13
views
No question representation invlolved in the equation of a QA reader
I have read Dense passage retrieval for Open Domain Question Answering, and in the page 7 they define the probability that a span contains the answer as follows:
Equation $(5)$ defines an attention ...
- 361
3
votes
1
answer
2k
views
What is the difference between the triplet loss and the contrastive loss?
What is the difference between the triplet loss and the contrastive loss?
They look same to me. I don't understand the nuances between the two. I have the following queries:
When to use what?
What ...
- 133
1
vote
2
answers
492
views
What should I think about when designing a custom loss function?
I'm trying to get my toy network to learn a sine wave.
I output (via tanh) a number between -1 and 1, and I want the network to minimise the following loss, where ...
- 113
1
vote
2
answers
138
views
What is the domain of the discriminator of a GAN?
I've read that the discriminator $D$ validates an image $D(x)$, where $x$ is either a real image or a fake one created by the generator, i.e. $ D(G(x))$.
What does the function of the discriminator ...
- 19
2
votes
0
answers
27
views
How to create a loss function that penalizes duplicate indices in the output tensor?
We're working on a sequence-to-sequence problem using pytorch, and are using cross-entropy to calculate the loss when comparing the output sequence to the target sequence. This works fine and ...
- 121
3
votes
1
answer
176
views
Why do we use "true labels" that are based on the output of our network in Deep Q-Learning?
In the original DQN paper, the $\ell_2$ loss is taken over the distance between our network output, $\hat{q}(s_j,a_j,w)$ and the labels $y_j=r_j+\gamma \cdot \max\limits_{a'} \hat{q}(s_{j+1},a',w^-)$, ...
- 341
1
vote
0
answers
41
views
Learning values in open ball: which final layers to employ?
I'm fairly new to deep learning and looking for some reference literature... Specifically, I want to train a neural network to predict vectors $v \in \mathbb{R}^3$ under the constraint $||v||\leq 1$.
...
- 111
0
votes
1
answer
51
views
How is catastrophic cancellation dealt with in loss functions?
It just occurred to me that this seems like it should be a very common problem that must have some kind of solution... Yet I'm not sure what it is...
If there is no solution, does this mean once a ...
- 296
0
votes
0
answers
32
views
how to define or calculate the similarity betweeen two curves as the loss funtion to optimize in the generative model?
I want to train a neural network as the curve productor that can generate the specific type of curves (e.g. exponential decay curves). I take the encoder-decoder structure, the curves in a dataset is ...
- 21
2
votes
1
answer
562
views
What is the reason we loop over epochs when training a neural network?
After reading through this thread and some other resources online, I still do not understand the role of epochs in training a neural network. I understand that one epoch is one iteration through the ...
- 21
0
votes
0
answers
55
views
How to fine-tune RoBERTa using Triplet Objective Function
I want to see if we can improve the triplet objective function of SBERT by slightly tweaking the equation terms. To do so,
In your opinion, what's the easiest way to fine-tune RoBERTa?
How can I ...
- 5
4
votes
2
answers
3k
views
What is the difference between a loss function and reward/penalty in Deep Reinforcement Learning?
In Deep Reinforcement Learning (DRL) I am having difficulties in understanding the difference between a Loss function, a reward/penalty and the integration of both in DRL.
Loss function: Given an ...
- 155
0
votes
0
answers
82
views
In CVAE's objective function, why do both terms condition on $\textbf{c}$?
I don't quite understand why, in Conditional Variational Autoencoder (CVAE), we concatenate a conditioning vector two times, at encoder and decoder respectively.
After we concatenate it once at the ...
- 277
0
votes
0
answers
57
views
How to reduce loss of Bi-LSTM handwriting recognition model?
I am currently training an bi-LSTM model which predicts the handwriting of an individual. I am hitting a current min loss of 1.2 and I think it is not a problem with the model because I copied a model ...
0
votes
1
answer
145
views
Does the summing or averaging of the weight gradients have anything to do with the cost function used?
I've been trying to implement my own neural network library and have been wondering if:
The SSE loss function includes the summation of the errors in the other training examples of the mini-batch (...
1
vote
1
answer
35
views
What inherent quality of a function makes it treated as either loss or evaluation metric?
A neural network model needs a loss function for training. The neural network needs to minimize the loss function.
A neural network is evaluated after training using a metric. The neural network needs ...
- 3,451
-1
votes
1
answer
139
views
Why is the cross-entropy a cost function?
The question looks foolish, but I think cross-entropy is somewhat weird as a cost function.
As a cost function for linear regression, the mean square error $ \sum_{i=1}^{n} (y_i - (ax_i+b)) ^2$ seems ...
- 125
3
votes
1
answer
84
views
Custom Tensorflow loss function that disincentivizes all black pixels
I'm training a Tensorflow model that receives an image and segments the image into foreground and background. That is, if the input image is w x h x 3, then the ...
- 33
1
vote
0
answers
57
views
a loss for binary step function data
I have some data with ground truth that looks like a binary step function, where part of it is 0 and part is one.
An example for the GT can be like ...
- 111
1
vote
0
answers
50
views
GANs: Why does iterative gradient descent sometimes optimise $\min_G \max_D V(D,G)$ and sometimes $\max_D \min_G V(D,G)$?
For the following minimax equation for generative adversarial networks (GANs),
$$\min_G \max_D V(D,G) = \mathbb{E}_{\boldsymbol{x}\sim p_{data}(\boldsymbol{x})}[\log D(\boldsymbol{x})] + \mathbb{E}_{\...
- 23
1
vote
1
answer
262
views
Why are logarithms used in GANs minimax equation?
The minimax equation for generative adversarial networks
$$\min_G \max_D V(D,G) = \mathbb{E}_{\boldsymbol{x}\sim p_{data}(\boldsymbol{x})}[\log D(\boldsymbol{x})] + \mathbb{E}_{\boldsymbol{z}\sim p_{\...
- 23
2
votes
2
answers
302
views
What specifically is the gradient of the log of the probability in policy gradient methods?
I am getting tripped up slightly by how specifically the gradient is calculated in policy gradient methods (just the intuitive understanding of it).
This Math Stack Exchange post is close, but I'm ...
- 151
1
vote
1
answer
342
views
Test accuracy decreases during my train process
I want to train a neural network model with the arcface loss function and try to combine it with domain adaption. But when the training process continues, I find the test accuracy first increases and ...
- 11
1
vote
0
answers
58
views
Where do the objective functions proposed in this paper by Carlini-Wagner attack come from?
I'm trying to understand the paper by Carlini and Wagner on deep neural networks adversarial attacks. On page 44, in Section V-A, it is explained how the loss function to the described problem was ...
- 111
2
votes
0
answers
35
views
Can a GIoU loss (generalized intersection over union) be used after an STN module (spatial transformer network)?
I have a model that uses an STN module for number detection and Mean Squared Error loss. But I would like to replace it for GIoU, because MSE doesn't take into account how much of the target area has ...
- 221
2
votes
1
answer
29
views
Are the domains of objective functions in AI always equals to $\mathbb{R}^D$ or subset of it?
Consider the following paragraph from the chapter named Vector Calculus from the textbook titled Mathematics for Machine Learning by Marc Peter Deisenroth et al.
Central to this chapter is the ...
- 3,451
0
votes
1
answer
73
views
Multi-class classification but a single feature sometimes boils it down to a binary-classification
I have a three-class classification problem for a large dataset. Classes are 0, 1, and 2. There's a categorical variable in my feature vectors such that when a sample point has this variable positive, ...
- 1