# Tag Info

### Should I choose a model with the smallest loss or highest accuracy?

You should choose the model A. The loss is just a differentiable proxy for accuracy. That said, the situation should be examined in more detail. If the higher loss is due to the data term, examine ...
Accepted

### What is the name of this letter $\mathcal{J}$?

It's an uppercase "J" from the math calligraphy alphabet, i.e. \mathcal{J} in latex. $\mathcal{J}$
Accepted

### Why L2 loss is more commonly used in Neural Networks than other loss functions?

I'll cover both L2 regularized loss, as well as Mean-Squared Error (MSE): MSE: L2 loss is continuously-differentiable across any domain, unlike L1 loss. This makes training more stable and allows ...
Accepted

### Should I choose a model with the smallest loss or highest accuracy?

You should note that both your results are consistent with a "true" probability of 87% accuracy, and your measurement of a difference between these models is not statistically significant. With an 87% ...
Accepted

### How contrastive loss work intuitively in siamese network

I think you are confused about why the margin exists. The margin exists in contrastive learning because we only want the model to output embeddings where negative samples are far from each other to a ...
Accepted

### How do I interpret this loss function?

This is the sum of squared residuals, and it uses notation from the mathematical subfield of linear algebra (arguably functional analysis). The double vertical bars indicate that we use the ...

### What is the domain of the discriminator of a GAN?

Formally, for an input $x$, $D(x)$ gives you the probability of $x$ being real. In this sense $D:\mathcal{X}\rightarrow [0,1]$, where $\mathcal{X}$ is the input space. That said, the output of the ...
Accepted

### PPO: policy loss becomes nan

You might want to try substituting the exponentiation with a piecewise-defined function that uses a numerical approximation that is more numerically stable for low values of the exponent, such as ...
Accepted

### Regression loss conditioned by the ground-truth values

Your suggestion should work to focus the ML more on larger angle examples. You may want to try a slightly simpler approach of weighting the loss (or the resulting gradient) by a factor depending on ...

### Is learning rate the only reason for training loss oscillation after few epochs?

The loss graph indicates that the model converged to a local minimum, already after a few epochs, and the weights start to oscillate around it. The learning rate is surely responsible for it, but it's ...
Accepted

### "Tweaking" the cost function to penalize rarer cases more severely

Yes, you can use weighted MSE by applying different coefficients to each data point. Here is an implementation given by Francisco Massa at https://discuss.pytorch.org/t/how-to-implement-weighted-mean-...

### What loss function will be correlated with classification metrics?

Different metrics measure different quantities, so there is no reason to expect different metrics to move together unless one is a function of the other (such as MSE and RMSE). Further, metrics like ...
Accepted

### Do we need to know or verify properties of loss functions / metrics' implementations?

Why we would like a function to satisfy some properties? If we're talking about a loss function, you need to prove at the very least that the function has a minimum, otherwise you can't expect it to ...
Accepted

### Val loss doesn’t decrease after a certain number of epochs

The training loss is continually decreasing, but the the validation loss shows a local minimum at epoch ~100 which indicates that there is overfitting starting to happen. This means that your model ...
Accepted

### Fluctuations in loss during in epoch evaluation of GRU

Oscillating loss is a symptom of divergent training: it can be due to large gradient updates, and/or numerical instability. Moreover, you said that it gets worse with higher LR: that's a confirm. To ...
Accepted

### What are the differences between loss surfaces that "derive"from different observations?

Let's take in consideration linear regression. You have a dataset composed by $x,y$ pairs, and you assume they are linearly related, thus you model this problem with LR: $$y = wx+b$$ Now, you want ...
1 vote

### How to use Categorical Cross Entropy for Multi-Label Classification?

You want to treat the problem as binary classification, and of course use binary cross-entropy. The only requirement is to ensure to use a sigmoid activation in your model's last layer.
1 vote
Accepted

### Can MSE be used for NN categorical classification problems

There are at least two reasons, why cross-entropy loss is preferred over mean squared error in classification problems. A theoretical reason Both aforementioned losses are negative logarithmic ...
1 vote
Accepted

### Non-Convex loss-surface although quadratic loss function

If I understood your question correctly - the quadratic loss function is not always convex. Its' convexity depends on the input it takes. For example, consider a very basic NN that takes some input $x$...
1 vote

### Why do we subtract logsumexp from the outputs of this neural network?

It's apparently for numerical stability. From the Wikipedia page for LogSumExp: A common purpose of using log-domain computations is to increase accuracy and avoid underflow and overflow problems ...
1 vote

### What is being optimized with WGAN loss? Is the generator maximizing or minimizing the critic value?

I think I understand what's happening with the loss functions now. Notation: D = discriminator/critic G = generator D(x) - Critic score on real data D(G(z)) - Critic score on fake data ∇_D - Critic ...
1 vote

### Visualizing the loss landscape in deep NN to compare optimization methods

How about: speed of convergence stability/variance w.r.t to the initial random seed (or other sources of variance like learning rate) presence/number of saddle points in your loss landscape
1 vote
Accepted

### What does IOU3 mean in this context?

Since we would like to distinguish among IoU values close to 1.0, we use IOU3 as the ground truth score for the SRN. It seems to be just IoU to the power of 3. They use the cube function because they ...
1 vote

### What does IOU3 mean in this context?

From context, I would say: Yes, it's IoU to the power of 3, since they want to have larger differences between values close to 1. Obviously, the difference between ...
1 vote
Accepted

### Does this modified version of the triplet loss function introduced with SBERT that uses the cosine similarity make sense?

A Loss function is just a function with a minimum. In machine learning though, we also require the loss function to be differentiable, otherwise no backpropagation and hence no weight updating. ...
1 vote

### Do smaller loss values during DQN training produce better policies?

I think it says something about the training progress, while another approach you can make sure is to look at the gradient norm. Sometimes, the training loss is really noisy while the gradient norm is ...
1 vote
Accepted

### How to explain peak in training history of a convolutional neural network?

I found that the peak was caused by the data I am using. Specifically, the MinMaxScaler changed the data shape and I resolved the issue by simply dividing to the max value.
1 vote

### Should I choose a model with the smallest loss or highest accuracy?

It depends on your application! Imagine a binary classifier that is always very "confident" - it always assigns P=100% to Class A and 0% to Class B, or vice versa (sometimes wrong, never uncertain!). ...
1 vote
Accepted

### How to handle invalid actions for next state in Q-learning loss

do we also want to consider the subset of invalid actions for the $\max\limits_{a}Q(s_{t+1},a)$ No. Doing so would go against the theory behind the Bellman equation from which the update derives. The ...
1 vote

### Where does the so-called 'loss' / 'loss function' fit into the idea of a perceptron / artificial neuron (as presented in the figure)?

Assume we have a binary classification problem, which we want to solve with a simple single-layer perceptron. For a 2d space, a perceptron will have 2 inputs $x_1$ and $x_2$, and a bias denoted $x_0$ ...

Only top scored, non community-wiki answers of a minimum length are eligible