Questions tagged [objective-functions]

For questions related to the concept of loss (or cost) function in the context of machine learning.

Filter by
Sorted by
Tagged with
0
votes
0answers
16 views

What loss function should be used for negative log likelihood labels

I am trying to build a ranking CNN model for document - query pairs using MS Marco dataset and python pytorch. My supervisor suggested to use the same CNN to extract feature vector for document and ...
1
vote
1answer
23 views

Test accuracy decreases during my train process

I want to train a neural network model with the arcface loss function and try to combine it with domain adaption. But when the training process continues, I find the test accuracy first increases and ...
1
vote
0answers
8 views

Can a GIoU loss (generalized intersection over union) be used after an STN module (spatial transformer network)?

I have a model that uses an STN module for number detection and Mean Squared Error loss. But I would like to replace it for GIoU, because MSE doesn't take into account how much of the target area has ...
1
vote
1answer
17 views

Are the domains of objective functions in AI always equals to $\mathbb{R}^D$ or subset of it?

Consider the following paragraph from the chapter named Vector Calculus from the textbook titled Mathematics for Machine Learning by Marc Peter Deisenroth et al. Central to this chapter is the ...
1
vote
1answer
17 views

Multi-class classification but a single feature sometimes boils it down to a binary-classification

I have a three-class classification problem for a large dataset. Classes are 0, 1, and 2. There's a categorical variable in my feature vectors such that when a sample point has this variable positive, ...
3
votes
0answers
29 views

How to choose the new layer and objective function for transfer learning on a neural network?

I have a base model $M$ trained on a data say type 1 for task $T$. Now, I want to update $M$ by applying transfer learning for it to work on data type 2 for the same task $T$. I am very new to AI/ML ...
2
votes
2answers
47 views

When would it make sense to perform a gradient descent step for each term of a loss function with multiple terms?

I am training a neural network using a mini-batch gradient descent algorithm. Now, consider the following loss function, which is composed of 2 terms. $$L = L_{\text{MSE}} + L_{\text{regularization}} \...
1
vote
0answers
45 views

What is the confusion loss for adversarial learning?

What is the confusion loss used in domain adaptation (DA) for adversarial learning/GANs? See this paper. Two domains: $s$: source domain $t$: target domain Generator/Discriminator setting: $M_s:x_s\...
0
votes
0answers
30 views
1
vote
0answers
31 views

When can we call a loss function "adaptive"?

A loss function is a measure of how bad our neural network is. We can decrease the loss by proper training. I came across the phrase "adaptive loss function" in several research papers. For ...
0
votes
0answers
30 views

Is there any significance for higher order gradients in artificial intelligence?

Although I don't know in detail, I am aware of the following facts regarding the use of gradients in some domains of artificial intelligence, especially in optimization. First order gradient: It ...
0
votes
0answers
16 views

Do I need to imagine loss curves of changing shapes in case of GANs?

Loss function, in general, is imagined as a curve in higher dimensional space with weights on input axes and loss on output axes. Suppose we have a neural network and we are training our neural ...
1
vote
1answer
82 views

Is there any difference between an objective function and a value function?

I found the usage of both objective function and value function in the same context. Context #1: In the paper titled Generative Adversarial Nets by Ian J. Goodfellow et al. We simultaneously train G ...
0
votes
1answer
44 views

Does average loss function in GAN training is just an approximation of value function and does not ensure convergence of generator and discriminator?

The value function on which convergence has been proved by the original paper of GAN is $$\min_G \max_DV(D, G) = \mathbb{E}_{x ∼ P_{data}}[\log D(x)] + \mathbb{E}_{z ∼ p_z}[log (1 - D(G(z)))]$$ and ...
2
votes
2answers
75 views

How to check whether my loss function is convex or not?

Loss functions are useful in calculating loss and then we can update the weights of a neural network. The loss function is thus useful in training neural networks. Consider the following excerpt from ...
3
votes
1answer
57 views

What are the necessary mathematical properties to be a loss function in gradient based optimizations?

Loss functions are used in training neural networks. I am interested in knowing the mathematical properties that are necessary for a loss function to participate in gradient descent optimization. I ...
0
votes
1answer
44 views

Why is the exponential loss used in this case?

I am reading a paper "Tracking-by-Segmentation With Online Gradient Boosting Decision Tree". In Section 2.1, the paper says I cannot understand the exponential loss function. In my opinion, ...
2
votes
0answers
32 views

Is optimizing weighted sum multi objective tasks considered a multi-task learning?

I have two sequence prediction tasks, finding $\vec{\pi} \in \Pi$ and $\vec{\psi} \in \Psi$. Each sequence has its own objective function, i.e. $f_1(\vec{\pi})$ and $f_2(\vec{\psi})$. The input for ...
1
vote
0answers
25 views

Can people set loss function of neural network by themselves instead of choosing cross entropy or mean square error?

I found people used deep neural network to get optimal policy by solving a nonconvex optimization problem. Moreover, they didn't use any set of training data and claimed that it's the difference ...
0
votes
0answers
35 views

An explanation involving the sign activation, its affect on the loss function, and the perceptron and perceptron criterion: what is this saying? (#2)

I recently asked a very similar question here, but the answer only seems to address the first part of the quote, rather than the second part that contains the perceptron criterion example. Therefore, ...
2
votes
0answers
30 views

Is the formula $\frac {1}{s}\sum _{j=1}^{s}|d_{j}-y_{j}(t)|$ the correct form of 0-1 loss function, in the context of Perceptron?

Per page 7 of this MIT lecture notes, the original single-layer Perceptron uses 0-1 loss function. Wikipedia uses $${\displaystyle {\frac {1}{s}}\sum _{j=1}^{s}|d_{j}-y_{j}(t)|} \tag{1}$$ to denote ...
0
votes
0answers
13 views

Loss function to Push response value towards extremes

I have a feature map whose values are in the range of [0,1]. I want to push these values either towards extreme 0 or 1 using some loss function. Since I don't have any target value so it had to be in ...
0
votes
1answer
58 views

An explanation involving the sign activation, its affect on the loss function, and the perceptron and perceptron criterion: what is this saying?

I am currently studying the textbook Neural Networks and Deep Learning by Charu C. Aggarwal. Chapter 1.2.1.3 Choice of Activation and Loss Functions says the following: The classical activation ...
1
vote
1answer
154 views

In logistic regression, why is the binary cross-entropy loss function convex?

I am studying logistic regression for binary classification. The loss function used is cross-entropy. For a given input $x$, if our model outputs $\hat{y}$ instead of $y$, the loss is given by $$\text{...
2
votes
3answers
244 views

Where does the so-called 'loss' / 'loss function' fit into the idea of a perceptron / artificial neuron (as presented in the figure)?

I am currently studying the textbook Neural Networks and Deep Learning by Charu C. Aggarwal. Chapter 1.2.1.3 Choice of Activation and Loss Functions presents the following figure: $\overline{X}$ is ...
0
votes
0answers
19 views

Good metrics and losses to use for Sequence-to-Sequence model for time-series prediction/forecasting

I am developing a sequence-to-sequence LSTM model for multi-step time series forecasting. I have the basic model working, so now I need to drill down on which loss function and evaluation metrics to ...
0
votes
0answers
19 views

Maximize delayed rewards

Given a Neural Network with a Dense(3) output and three actions: 'B' is [0, 0, 1] (= 1, for the sake of our example) 'N' is ...
3
votes
0answers
54 views

Loss function to minimize the distance between sets

Are there references or links to examples about loss functions "Distance Metrics" which could be used to minimize the distance between two sets for a neural network. More precisely, this ...
0
votes
0answers
26 views

Problems while transforming a 2D Variational Autoencoder into a 1D Version

I am trying to addapt the Keras variational autoencoder (VAE) here from a 2-D input/output (matrix of a picture) to a 1-D input/output (just a vector). I thought this would be a fearly easy task, but ...
0
votes
1answer
43 views

Which loss function could I use to solve a regression problem as a classification problem (where we discretize the labels into buckets)?

I am considering a rather typical regression problem, but, for practice, I am trying to implement this as a classification problem. The setup is as follows. I have $\mathbb{R}$-valued labels $y_i \in [...
0
votes
0answers
29 views

How to improve the Loss and Learning curves and smoothen them

I am fairly new to deep learning and I have been testing out several architectures for the segmentation task of clouds in satellite imagery. I am using a simple Unet as my benchmark, Unet++, Efficient ...
0
votes
0answers
19 views

Defining optimal false-positives and false-negatives balance with a cost function

My attempt to solve the problem below: $$\text{cost function} = C = (TP \cdot CTP) + (FN \cdot CFN) + (FP \cdot CFP) + (TN \cdot CTN) = ((1 - (1 - FP)^2) \cdot 1000) + (FN \cdot CFN) + (FP \cdot ...
0
votes
0answers
44 views

How is it possible that the softmax combined with the MSE in a molecule classification task performs than than the cross-entropy?

I'm working on a GNN project associated with molecule classification. The project is to classify if the atom in the molecule will initiate a certain reaction. For example, a molecule can be ...
0
votes
1answer
125 views

Why is loss displayed as a parabola in mean squared error with gradient descent?

I'm looking at the loss function: mean squared error with gradient descent in machine learning. I'm building a single-neuron network (perceptron) that outputs a linear number. For example: Input * ...
2
votes
0answers
36 views

Could the inputs of the mean squared-error loss function be transformed to allow larger learning rates?

In the context of a neural network $\hat{y} = f_\theta(\mathbf{x})$ with parameters $\theta$ that is trained to perform regression such that the prediction $\hat{\mathbf{y}} = [\hat{y}_1,\hat{y}_2,...,...
0
votes
2answers
114 views

In variational autoencoders, why do people use MSE for the loss?

In VAEs, we try to maximize the ELBO $\mathbb(E_q log\ p(x|z) + D_{KL}(q(z|x), p(z))$), but I see that many implement the first term as MSE of the image and it's reconstruction. Is this mathematically ...
3
votes
1answer
65 views

Is it possible to use an internal layer's outputs in a loss function?

For a network of the form: Input(10) Dense(200) Dense(100+10) Dense(20) Output() Those +10 outputs are what I want to add to ...
1
vote
1answer
51 views

Is the policy gradient expression in Fundamentals of Deep Learning wrong?

I don't understand the policy gradient as explained in Chapter-9 (Deep Reinforcement Learning) of the book Fundamentals of deep learning. Here is the whole paragraph: Policy Learning via Policy ...
1
vote
1answer
103 views

Why does the implementation of REINFORCE algorithm minimize the gradient term but not the loss?

I read the book "Foundation of Deep Reinforcement Learning, Laura Graesser and Wah Loon Keng", and when I go through the REINFORCE algorithm, they show the objective function: $$ J\left(\...
2
votes
0answers
19 views

Is it normal getting noise values in the error history along training iteration?

I'm giving my first steps in really learning machine learning. As an exercise in my online course, it was asked for me to code the Cost function of some neural network that should resolve the ...
1
vote
0answers
18 views

Do dataset sizes matter in a Style GAN?

When working with classifiers, a class imbalance is a huge issue for our models. If we have too many images of class 1 and too few images from ...
2
votes
1answer
99 views

How to deal with losses on different scales in multi-task learning?

Say I'm training a model for multiple tasks by trying to minimize sum of losses $L_1 + L_2$ via gradient descent. If these losses are on a different scale, the one whose range is greater will dominate ...
1
vote
0answers
13 views

How to update all the weights in case only one data out of n signals is observable

If we have cost function as $$E_i = (D_i -Y_i)^T Q (D_i -Y_i)$$, where $$Q=\begin{bmatrix} 1 & 0 & 0\\ 0 & 0 & 0\\ 0 & 0 & 0 \end{bmatrix}$$( in case only one data signal can ...
0
votes
0answers
23 views

Gradient of CTC Loss?

I am having a hard time figuring out how the gradient of the CTC loss function looks like. Could anyone explain that to me?
0
votes
1answer
26 views

setting up last layer in tensoflow for class type of label [closed]

I am creating a NN in tensorflow keras. the inputs are all float and the output is a class. The output currently encoded as a float, but only has 4 values (0,1,2,3). My model is similar to this: ...
1
vote
0answers
60 views

BlackOut - ICLR 2016: need help understanding the cost function derivative

In the ICLR 2016 paper BlackOut: Speeding up Recurrent Neural Network Language Models with very Large Vocabularies, on page 3, for eq. 4: $$ J_{ml}^s(\theta) = log \ p_{\theta}(w_i | s) $$ They have ...
0
votes
1answer
53 views

What error should I use for RNN?

I'm relatively new to machine learning, and I don't know what error I should use for an RNN. I want to use a simple Elman RNN to predict the cases of Covid-19 there will be in a hospital for the next ...
2
votes
0answers
32 views

Single-value loss/training in a CNN with a tensor output

I am playing around with an idea of using using Q-learning with a DQN (Deep Q-Network), to determine the optimal position of a number of 'units' on a grid of allowed locations, according to some ...
3
votes
1answer
93 views

How to incorporate a symmetry constraint in the loss function to train a CNN?

I have a task of extremely sparse binary segmentation, i.e. the segmentation mask contains either 0 or 1, and there are ~95% zeros and only ~5% ones. I use the focal loss to address the sparseness (...
1
vote
0answers
24 views

1
2 3 4 5