Tag Info

Should I choose a model with the smallest loss or highest accuracy?

You should choose the model A. The loss is just a differentiable proxy for accuracy. That said, the situation should be examined in more detail. If the higher loss is due to the data term, examine ...
• 469
Accepted

Why L2 loss is more commonly used in Neural Networks than other loss functions?

I'll cover both L2 regularized loss, as well as Mean-Squared Error (MSE): MSE: L2 loss is continuously-differentiable across any domain, unlike L1 loss. This makes training more stable and allows ...
• 191
Accepted

What is the name of this letter $\mathcal{J}$?

It's an uppercase "J" from the math calligraphy alphabet, i.e. \mathcal{J} in latex. $\mathcal{J}$
• 876
Accepted

Should I choose a model with the smallest loss or highest accuracy?

You should note that both your results are consistent with a "true" probability of 87% accuracy, and your measurement of a difference between these models is not statistically significant. With an 87% ...
• 23.8k
Accepted

How contrastive loss work intuitively in siamese network

I think you are confused about why the margin exists. The margin exists in contrastive learning because we only want the model to output embeddings where negative samples are far from each other to a ...
• 271

Is learning rate the only reason for training loss oscillation after few epochs?

The loss graph indicates that the model converged to a local minimum, already after a few epochs, and the weights start to oscillate around it. The learning rate is surely responsible for it, but it's ...
• 3,503

What is the domain of the discriminator of a GAN?

Formally, for an input $x$, $D(x)$ gives you the probability of $x$ being real. In this sense $D:\mathcal{X}\rightarrow [0,1]$, where $\mathcal{X}$ is the input space. That said, the output of the ...
Accepted

PPO: policy loss becomes nan

You might want to try substituting the exponentiation with a piecewise-defined function that uses a numerical approximation that is more numerically stable for low values of the exponent, such as ...
• 66
1 vote

What does IOU3 mean in this context?

Since we would like to distinguish among IoU values close to 1.0, we use IOU3 as the ground truth score for the SRN. It seems to be just IoU to the power of 3. They use the cube function because they ...
• 121
1 vote

What does IOU3 mean in this context?

From context, I would say: Yes, it's IoU to the power of 3, since they want to have larger differences between values close to 1. Obviously, the difference between ...
• 175
1 vote
Accepted

Does this modified version of the triplet loss function introduced with SBERT that uses the cosine similarity make sense?

A Loss function is just a function with a minimum. In machine learning though, we also require the loss function to be differentiable, otherwise no backpropagation and hence no weight updating. ...
• 3,503
1 vote
Accepted

How to explain peak in training history of a convolutional neural network?

I found that the peak was caused by the data I am using. Specifically, the MinMaxScaler changed the data shape and I resolved the issue by simply dividing to the max value.
• 249
1 vote

Should I choose a model with the smallest loss or highest accuracy?

It depends on your application! Imagine a binary classifier that is always very "confident" - it always assigns P=100% to Class A and 0% to Class B, or vice versa (sometimes wrong, never uncertain!). ...
• 317
1 vote
Accepted

How to handle invalid actions for next state in Q-learning loss

do we also want to consider the subset of invalid actions for the $\max\limits_{a}Q(s_{t+1},a)$ No. Doing so would go against the theory behind the Bellman equation from which the update derives. The ...
• 23.8k
1 vote

Where does the so-called 'loss' / 'loss function' fit into the idea of a perceptron / artificial neuron (as presented in the figure)?

Assume we have a binary classification problem, which we want to solve with a simple single-layer perceptron. For a 2d space, a perceptron will have 2 inputs $x_1$ and $x_2$, and a bias denoted $x_0$ ...
• 732
1 vote

Where does the so-called 'loss' / 'loss function' fit into the idea of a perceptron / artificial neuron (as presented in the figure)?

The loss function is simply a way to measure how wrong a neural network is, it doesn't affect the output of the neuron. Say we have a neural network with 3 output neurons that attempts to classify ...
• 11
1 vote
Accepted

Where does the so-called 'loss' / 'loss function' fit into the idea of a perceptron / artificial neuron (as presented in the figure)?

Loss function is a function used to measure the loss. It is not used in any component of a neuron. It is used in updating the weights of the neuron i.e., in order to train the neuron. The contribution ...
• 3,099
1 vote

Is it okay to calculate the validation loss over batches instead of the whole validation set for speed purposes?

I assume you intended to write compute the evaluation metric over the validation set in batches; you do not compute loss over the validation set! That is quite a standard practice in many academic ...
• 151
1 vote

What is the "contradictory loss" in the "Old Photo Restoration via Deep Latent Space Translation" paper?

The contradictory loss is the same loss function that the discriminator would normally use, except with deliberately incorrect labels. That is, when you train the generator, the output of the ...
• 23.8k
1 vote

Why does the accuracy drop while the loss decrease, as the number of epochs increases?

Decrease of loss does not essentially lead to increase of accuracy (most of the time it happens but sometime it may not happen). To know why, you can have a look at this question. The network cares ...
• 430

Only top scored, non community-wiki answers of a minimum length are eligible