6
votes
Should I choose a model with the smallest loss or highest accuracy?
You should choose the model A. The loss is just a differentiable proxy for accuracy.
That said, the situation should be examined in more detail. If the higher loss is due to the data term, examine ...
- 489
5
votes
Accepted
What is the name of this letter $\mathcal{J}$?
It's an uppercase "J" from the math calligraphy alphabet, i.e. \mathcal{J} in latex.
$\mathcal{J}$
- 1,201
5
votes
Accepted
Why L2 loss is more commonly used in Neural Networks than other loss functions?
I'll cover both L2 regularized loss, as well as Mean-Squared Error (MSE):
MSE:
L2 loss is continuously-differentiable across any domain, unlike L1 loss. This makes training more stable and allows ...
- 191
4
votes
Accepted
Should I choose a model with the smallest loss or highest accuracy?
You should note that both your results are consistent with a "true" probability of 87% accuracy, and your measurement of a difference between these models is not statistically significant. With an 87% ...
- 30.2k
4
votes
Accepted
How contrastive loss work intuitively in siamese network
I think you are confused about why the margin exists.
The margin exists in contrastive learning because we only want the model to output embeddings where negative samples are far from each other to a ...
- 319
4
votes
Accepted
How do I interpret this loss function?
This is the sum of squared residuals, and it uses notation from the mathematical subfield of linear algebra (arguably functional analysis).
The double vertical bars indicate that we use the ...
- 548
3
votes
What is the domain of the discriminator of a GAN?
Formally, for an input $x$, $D(x)$ gives you the probability of $x$ being real. In this sense $D:\mathcal{X}\rightarrow [0,1]$, where $\mathcal{X}$ is the input space.
That said, the output of the ...
3
votes
Accepted
PPO: policy loss becomes nan
You might want to try substituting the exponentiation with a piecewise-defined function that uses a numerical approximation that is more numerically stable for low values of the exponent, such as ...
- 106
3
votes
Accepted
Regression loss conditioned by the ground-truth values
Your suggestion should work to focus the ML more on larger angle examples.
You may want to try a slightly simpler approach of weighting the loss (or the resulting gradient) by a factor depending on ...
- 30.2k
2
votes
Is learning rate the only reason for training loss oscillation after few epochs?
The loss graph indicates that the model converged to a local minimum, already after a few epochs, and the weights start to oscillate around it. The learning rate is surely responsible for it, but it's ...
- 5,218
2
votes
Accepted
"Tweaking" the cost function to penalize rarer cases more severely
Yes, you can use weighted MSE by applying different coefficients to each data point.
Here is an implementation given by Francisco Massa at https://discuss.pytorch.org/t/how-to-implement-weighted-mean-...
2
votes
What loss function will be correlated with classification metrics?
Different metrics measure different quantities, so there is no reason to expect different metrics to move together unless one is a function of the other (such as MSE and RMSE).
Further, metrics like ...
- 548
2
votes
Accepted
Do we need to know or verify properties of loss functions / metrics' implementations?
Why we would like a function to satisfy some properties?
If we're talking about a loss function, you need to prove at the very least that the function has a minimum, otherwise you can't expect it to ...
- 5,218
2
votes
Accepted
Val loss doesn’t decrease after a certain number of epochs
The training loss is continually decreasing, but the the validation loss shows a local minimum at epoch ~100 which indicates that there is overfitting starting to happen. This means that your model ...
- 922
2
votes
Accepted
Fluctuations in loss during in epoch evaluation of GRU
Oscillating loss is a symptom of divergent training: it can be due to large gradient updates, and/or numerical instability. Moreover, you said that it gets worse with higher LR: that's a confirm.
To ...
- 2,443
2
votes
Accepted
What are the differences between loss surfaces that "derive"from different observations?
Let's take in consideration linear regression. You have a dataset composed by $x,y$ pairs, and you assume they are linearly related, thus you model this problem with LR:
$$
y = wx+b
$$
Now, you want ...
- 1,308
1
vote
How to use Categorical Cross Entropy for Multi-Label Classification?
You want to treat the problem as binary classification, and of course use binary cross-entropy.
The only requirement is to ensure to use a sigmoid activation in your model's last layer.
- 5,218
1
vote
Accepted
Can MSE be used for NN categorical classification problems
There are at least two reasons, why cross-entropy loss is preferred over mean squared error in classification problems.
A theoretical reason
Both aforementioned losses are negative logarithmic ...
1
vote
Accepted
Non-Convex loss-surface although quadratic loss function
If I understood your question correctly - the quadratic loss function is not always convex. Its' convexity depends on the input it takes. For example, consider a very basic NN that takes some input $x$...
- 371
1
vote
Why do we subtract logsumexp from the outputs of this neural network?
It's apparently for numerical stability. From the Wikipedia page for LogSumExp:
A common purpose of using log-domain computations is to increase accuracy and avoid underflow and overflow problems ...
- 111
1
vote
What is being optimized with WGAN loss? Is the generator maximizing or minimizing the critic value?
I think I understand what's happening with the loss functions now.
Notation:
D = discriminator/critic
G = generator
D(x) - Critic score on real data
D(G(z)) - Critic score on fake data
∇_D - Critic ...
1
vote
Visualizing the loss landscape in deep NN to compare optimization methods
How about:
speed of convergence
stability/variance w.r.t to the initial random seed (or other sources of variance like learning rate)
presence/number of saddle points in your loss landscape
- 76
1
vote
Accepted
What does IOU3 mean in this context?
Since we would like to distinguish among IoU values close to 1.0, we
use IOU3 as the ground truth score for the SRN.
It seems to be just IoU to the power of 3. They use the cube function because they ...
- 181
1
vote
What does IOU3 mean in this context?
From context, I would say: Yes, it's IoU to the power of 3, since they want to have larger differences between values close to 1. Obviously, the difference between ...
- 249
1
vote
Accepted
Does this modified version of the triplet loss function introduced with SBERT that uses the cosine similarity make sense?
A Loss function is just a function with a minimum.
In machine learning though, we also require the loss function to be differentiable, otherwise no backpropagation and hence no weight updating. ...
- 5,218
1
vote
Do smaller loss values during DQN training produce better policies?
I think it says something about the training progress, while another approach you can make sure is to look at the gradient norm. Sometimes, the training loss is really noisy while the gradient norm is ...
1
vote
Accepted
How to explain peak in training history of a convolutional neural network?
I found that the peak was caused by the data I am using. Specifically, the MinMaxScaler changed the data shape and I resolved the issue by simply dividing to the max value.
- 249
1
vote
Should I choose a model with the smallest loss or highest accuracy?
It depends on your application! Imagine a binary classifier that is always very "confident" - it always assigns P=100% to Class A and 0% to Class B, or vice versa (sometimes wrong, never uncertain!). ...
- 337
1
vote
Accepted
How to handle invalid actions for next state in Q-learning loss
do we also want to consider the subset of invalid actions for the $\max\limits_{a}Q(s_{t+1},a)$
No.
Doing so would go against the theory behind the Bellman equation from which the update derives. The ...
- 30.2k
1
vote
Where does the so-called 'loss' / 'loss function' fit into the idea of a perceptron / artificial neuron (as presented in the figure)?
Assume we have a binary classification problem, which we want to solve with a simple single-layer perceptron. For a 2d space, a perceptron will have 2 inputs $x_1$ and $x_2$, and a bias denoted $x_0$ ...
- 927
Only top scored, non community-wiki answers of a minimum length are eligible