# Tag Info

Accepted

### In which cases is the categorical cross-entropy better than the mean squared error?

As a rule of thumb, mean squared error (MSE) is more appropriate for regression problems, that is, problems where the output is a numerical value (i.e. a floating-point number or, in general, a real ...
• 34.9k
Accepted

• 121
1 vote
Accepted

### In variational autoencoders, why do people use MSE for the loss?

If $p(x|z) \sim \mathcal{N}(f(z), I)$, then \begin{align} \log\ p(x|z) &\sim \log\ \exp(-(x-f(z))^2) \\ &\sim -(x-f(z))^2 \\ &= -(x-\hat{x})^2, \end{align} where $\hat{x}$, the ...
• 139
1 vote
Accepted

### What error should I use for RNN?

To provide a good answer would fill several pages. To keep it very simple try many different loss functions on your model. Your goal is to have the highest performance based on some desired ...
• 1,503
1 vote
Accepted

### Would either $L_1$ or $L_2$ regularisation lower the MSE on the training and test data?

The answer is largely the same whether we consider $\ell_1$ or $\ell_2$ regularisation, so I will just speak generally about regularisation. Mean square error for training data Given some training ...
• 940
1 vote
Accepted

### What is the meaning of these equations in Noise2Noise paper?

The equation you are referring to is called Mean Squared Error (or $L_2$ loss) and it is used for regression tasks, where the goal is to predict a real value given some input. In your case, the inputs ...
• 600
1 vote
Accepted

### What is the definition of a loss function in the context of neural networks?

A loss function is what helps you "train" your neural network to do what you want it to do. A better way to word it to begin with would be an "objective" function. This function ...
• 156
1 vote

### What is the advantage of using cross entropy loss & softmax?

Short answer: larger gradients That is not the derivative of the softmax function. $t - o$ is the combined derivative of the softmax function and cross entropy loss. Cross entropy loss is used to ...
• 470
1 vote

### Why do we calculate the mean squared error loss to improve the value approximation in Advantage Actor-Critic Algorithm?

I believe that the author is referring to how the networks are trained in Deep RL. Consider Deep Q-Learning where the $Q(s,a)$ is approximated using a neural network. Then the loss function used to ...
• 4,198
1 vote

### Is it normal to have the root mean squared error greater on the test dataset than on the training dataset?

It is common to have root mean squared error (RMSE) greater on the test dataset than on the training dataset (this is equal to having accuracy/score higher for model in training dataset than test ...
1 vote

### How to determine the target value when using ReLU as activation function?

You are misunderstanding something. You are mixing up inner layers with the output layer. But the question was very good. Fist of all, with the only one layer and one neuron neural networks it does ...
• 91
1 vote

### How to determine the target value when using ReLU as activation function?

ReLU and sigmoid have different properties (i.e. range), as you already noticed. I've never seen the ReLU being used as the activation function of the output layer (but some people may use it for some ...
• 34.9k
1 vote

### How to express accuracy of a regression ANN that uses MSE loss function?

Just as a general remark, notice that technically we don't use the term "accuracy" for regression settings, such as yours - only for classification ones. If RMSE is 'in the units of the quantity ...
• 985
1 vote
Accepted

### What are the advantages of the Kullback-Leibler over the MSE/RMSE?

In the context of Variational Inference (VI): the KL allows you to move from the unknown posterior $p(z \mid x)$, to the known joint $p(z,x)=p(x|z)p(z)$ and optimize only the ELBO. You cannot do this ...
1 vote

### Is it a good idea to train a CNN to detect the hydration value (percentage) in skin images and evaluate it with the MSE?

Your initial idea seems about right. Before creating your own classifier you might want to try transfer learning, using some pretrained network like VGG16 that is incorporated in most of machine ...
• 333

Only top scored, non community-wiki answers of a minimum length are eligible