Skip to main content
Share Your Experience: Take the 2024 Developer Survey
6 votes

Should I choose a model with the smallest loss or highest accuracy?

You should choose the model A. The loss is just a differentiable proxy for accuracy. That said, the situation should be examined in more detail. If the higher loss is due to the data term, examine ...
ssegvic's user avatar
  • 499
5 votes
Accepted

What is the name of this letter $\mathcal{J}$?

It's an uppercase "J" from the math calligraphy alphabet, i.e. \mathcal{J} in latex. $\mathcal{J}$
Taw's user avatar
  • 1,261
5 votes
Accepted

Why L2 loss is more commonly used in Neural Networks than other loss functions?

I'll cover both L2 regularized loss, as well as Mean-Squared Error (MSE): MSE: L2 loss is continuously-differentiable across any domain, unlike L1 loss. This makes training more stable and allows ...
Ryan S's user avatar
  • 191
4 votes
Accepted

Should I choose a model with the smallest loss or highest accuracy?

You should note that both your results are consistent with a "true" probability of 87% accuracy, and your measurement of a difference between these models is not statistically significant. With an 87% ...
Neil Slater's user avatar
  • 32.5k
4 votes
Accepted

How contrastive loss work intuitively in siamese network

I think you are confused about why the margin exists. The margin exists in contrastive learning because we only want the model to output embeddings where negative samples are far from each other to a ...
DKDK's user avatar
  • 329
4 votes
Accepted

How do I interpret this loss function?

This is the sum of squared residuals, and it uses notation from the mathematical subfield of linear algebra (arguably functional analysis). The double vertical bars indicate that we use the ...
Dave's user avatar
  • 618
3 votes

What is the domain of the discriminator of a GAN?

Formally, for an input $x$, $D(x)$ gives you the probability of $x$ being real. In this sense $D:\mathcal{X}\rightarrow [0,1]$, where $\mathcal{X}$ is the input space. That said, the output of the ...
Eduardo Montesuma's user avatar
3 votes
Accepted

PPO: policy loss becomes nan

You might want to try substituting the exponentiation with a piecewise-defined function that uses a numerical approximation that is more numerically stable for low values of the exponent, such as ...
kmf's user avatar
  • 106
3 votes
Accepted

Regression loss conditioned by the ground-truth values

Your suggestion should work to focus the ML more on larger angle examples. You may want to try a slightly simpler approach of weighting the loss (or the resulting gradient) by a factor depending on ...
Neil Slater's user avatar
  • 32.5k
2 votes

Is learning rate the only reason for training loss oscillation after few epochs?

The loss graph indicates that the model converged to a local minimum, already after a few epochs, and the weights start to oscillate around it. The learning rate is surely responsible for it, but it's ...
Edoardo Guerriero's user avatar
2 votes
Accepted

"Tweaking" the cost function to penalize rarer cases more severely

Yes, you can use weighted MSE by applying different coefficients to each data point. Here is an implementation given by Francisco Massa at https://discuss.pytorch.org/t/how-to-implement-weighted-mean-...
Jaume Oliver Lafont's user avatar
2 votes
Accepted

Why do we subtract logsumexp from the outputs of this neural network?

Though I may be answering this question quite late, I would like to address your inquiry (because it's possible that other individuals may also have similar queries. :D) You can think of the last 3 ...
present42's user avatar
2 votes
Accepted

Do we need to know or verify properties of loss functions / metrics' implementations?

Why we would like a function to satisfy some properties? If we're talking about a loss function, you need to prove at the very least that the function has a minimum, otherwise you can't expect it to ...
Edoardo Guerriero's user avatar
2 votes

What loss function will be correlated with classification metrics?

Different metrics measure different quantities, so there is no reason to expect different metrics to move together unless one is a function of the other (such as MSE and RMSE). Further, metrics like ...
Dave's user avatar
  • 618
2 votes
Accepted

Val loss doesn’t decrease after a certain number of epochs

The training loss is continually decreasing, but the the validation loss shows a local minimum at epoch ~100 which indicates that there is overfitting starting to happen. This means that your model ...
Snehal Patel's user avatar
2 votes
Accepted

Fluctuations in loss during in epoch evaluation of GRU

Oscillating loss is a symptom of divergent training: it can be due to large gradient updates, and/or numerical instability. Moreover, you said that it gets worse with higher LR: that's a confirm. To ...
Luca Anzalone's user avatar
2 votes
Accepted

What are the differences between loss surfaces that "derive"from different observations?

Let's take in consideration linear regression. You have a dataset composed by $x,y$ pairs, and you assume they are linearly related, thus you model this problem with LR: $$ y = wx+b $$ Now, you want ...
Alberto's user avatar
  • 2,173
2 votes
Accepted

Loss goes down but never below a certain treshold

In the first example, Sigmoid activation cannot go below 0. Either modify your encoding or switch to a different activation (tanh for example can go to -1). On the second issue, it is customary to ...
foreverska's user avatar
  • 1,288
1 vote
Accepted

Can MSE be used for NN categorical classification problems

There are at least two reasons, why cross-entropy loss is preferred over mean squared error in classification problems. A theoretical reason Both aforementioned losses are negative logarithmic ...
Vladimir Yankovskiy's user avatar
1 vote
Accepted

Non-Convex loss-surface although quadratic loss function

If I understood your question correctly - the quadratic loss function is not always convex. Its' convexity depends on the input it takes. For example, consider a very basic NN that takes some input $x$...
Hadar Sharvit's user avatar
1 vote

Why do we subtract logsumexp from the outputs of this neural network?

It's apparently for numerical stability. From the Wikipedia page for LogSumExp: A common purpose of using log-domain computations is to increase accuracy and avoid underflow and overflow problems ...
joe's user avatar
  • 111
1 vote

What is being optimized with WGAN loss? Is the generator maximizing or minimizing the critic value?

I think I understand what's happening with the loss functions now. Notation: D = discriminator/critic G = generator D(x) - Critic score on real data D(G(z)) - Critic score on fake data ∇_D - Critic ...
Gabriel Mongaras's user avatar
1 vote

Visualizing the loss landscape in deep NN to compare optimization methods

How about: speed of convergence stability/variance w.r.t to the initial random seed (or other sources of variance like learning rate) presence/number of saddle points in your loss landscape
ImotVoksim's user avatar
1 vote
Accepted

What does IOU3 mean in this context?

Since we would like to distinguish among IoU values close to 1.0, we use IOU3 as the ground truth score for the SRN. It seems to be just IoU to the power of 3. They use the cube function because they ...
sfotiadis's user avatar
  • 291
1 vote

What does IOU3 mean in this context?

From context, I would say: Yes, it's IoU to the power of 3, since they want to have larger differences between values close to 1. Obviously, the difference between ...
Kroshtan's user avatar
  • 259
1 vote
Accepted

Does this modified version of the triplet loss function introduced with SBERT that uses the cosine similarity make sense?

A Loss function is just a function with a minimum. In machine learning though, we also require the loss function to be differentiable, otherwise no backpropagation and hence no weight updating. ...
Edoardo Guerriero's user avatar
1 vote

Do smaller loss values during DQN training produce better policies?

I think it says something about the training progress, while another approach you can make sure is to look at the gradient norm. Sometimes, the training loss is really noisy while the gradient norm is ...
DongDong Chen's user avatar
1 vote
Accepted

How to explain peak in training history of a convolutional neural network?

I found that the peak was caused by the data I am using. Specifically, the MinMaxScaler changed the data shape and I resolved the issue by simply dividing to the max value.
Cristian M's user avatar
1 vote

Should I choose a model with the smallest loss or highest accuracy?

It depends on your application! Imagine a binary classifier that is always very "confident" - it always assigns P=100% to Class A and 0% to Class B, or vice versa (sometimes wrong, never uncertain!). ...
Edward Dixon's user avatar
1 vote
Accepted

How to handle invalid actions for next state in Q-learning loss

do we also want to consider the subset of invalid actions for the $\max\limits_{a}Q(s_{t+1},a)$ No. Doing so would go against the theory behind the Bellman equation from which the update derives. The ...
Neil Slater's user avatar
  • 32.5k

Only top scored, non community-wiki answers of a minimum length are eligible