6

As a rule of thumb, mean squared error (MSE) is more appropriate for regression problems, that is, problems where the output is a numerical value (i.e. a floating-point number or, in general, a real number). However, in principle, you can use the MSE for classification problems too (even though that may not be a good idea). MSE can be preceded by the ...


3

In a classification problem it's better to get higher error and higher error slope when we predict the label wrong. As you see in the graph by using cross-entropy you get high error when the algorithm predict a label wrong and small error when the prediacted label is close enough, so it helps us to separate the predicted classes better.


2

You can not use error to reliably measure accuracy. Error is best used as a measure of how fast the model is currently learning. As an example, using different loss functions (cross entorpy vs MSE) results in massively different values for the error at similar accuracy. Also considering this, an error of 0.0000000001 quite often has lower validation set ...


2

Accuracy itself isn't a sufficient way to compare two models. For example, you need to consider the precision and recall stats (see confusion matrix) and calculate some other metrics like f1 score. The measurement of accuracy is only the initial state that helps us to know if a model is "working". But in order to understand and compare you need to know how ...


2

MSE just measures the squared difference between actual and target values. It can still correctly classify the values, but perhaps not with the same confidence - leading to a higher loss (e.g. an output of 0.77 vs 0.98 when the target is 1). In terms of which is better, I wouldn't know without the specifics of your problem. It is possible the higher loss ...


2

This is very easy to prove. Let's first prove that, if $\hat{y}_k = y_k$, then the $E = 0$. I will leave all steps, so that it's super clear. \begin{align} E &=\frac{1}{2}\sum_k(\hat{y}_k - y_k)^2 \\ &=\frac{1}{2}\sum_k(y_k - y_k)^2\\ &=\frac{1}{2}\sum_k(0)^2\\ &=\frac{1}{2}\sum_k 0\\ &=\frac{1}{2} 0\\ &=0\\ \end{align} To prove the ...


2

We sometimes see that binary cross-entropy (BCE) loss is used for regression problems. This post is my opinion on using BCE for regression problems. The figure below is the plots of BCE, $-t*\log(x) - (1-t)*\log(1-x)$, for several target values $t = 0.0, 0.1, ..., 0.5$. (The plots for $t>0.5$ are mirror images of those for $t<0.5$, so I omitted them.) ...


2

On page 5 of the VAE paper, it's clearly stated We let $p_{\boldsymbol{\theta}}(\mathbf{x} \mid \mathbf{z})$ be a multivariate Gaussian (in case of real-valued data) or Bernoulli (in case of binary data) whose distribution parameters are computed from $\mathbf{z}$ with a MLP (a fully-connected neural network with a single hidden layer, see appendix $\mathrm{...


1

To provide a good answer would fill several pages. To keep it very simple try many different loss functions on your model. Your goal is to have the highest performance based on some desired prediction metric (e.g., RMSE, MAE, MAPE, etc.). You almost always have plenty of time to try many loss functions so you don't need to have a full understanding, and ...


1

The answer is largely the same whether we consider $\ell_1$ or $\ell_2$ regularisation, so I will just speak generally about regularisation. Mean square error for training data Given some training data $\{(x_i, y_i)\}_{i = 1}^n$, a linear regression line $Y = aX + b$ fit using the least squares method looks for coefficients that minimise the sum of squares, ...


1

The equation you are referring to is called Mean Squared Error (or $L_2$ loss) and it is used for regression tasks, where the goal is to predict a real value given some input. In your case, the inputs are measurements of temperature $y$, either at a certain point in time or point in space or both or none, this is not clear from the image. Now, the goal would ...


1

Short answer: larger gradients That is not the derivative of the softmax function. $t - o$ is the combined derivative of the softmax function and cross entropy loss. Cross entropy loss is used to simplify the derivative of the softmax function. In the end, you do end up with a different gradients. It would be like if you ignored the sigmoid derivative when ...


1

I believe that the author is referring to how the networks are trained in Deep RL. Consider Deep Q-Learning where the $Q(s,a)$ is approximated using a neural network. Then the loss function used to train the network is $$\mathbb{E}[(r + \gamma \max_{a'} Q(s',a') - Q(s,a))^2]\;.$$ Here, $r + \gamma \max_{a'} Q(s',a')$ is your target, what you want your ...


1

It is common to have root mean squared error (RMSE) greater on the test dataset than on the training dataset (this is equal to having accuracy/score higher for model in training dataset than test dataset). This normally happens because the training data are assesed on the same data that have been learnt before, while the test dataset may have data that are ...


1

You are misunderstanding something. You are mixing up inner layers with the output layer. But the question was very good. Fist of all, with the only one layer and one neuron neural networks it does not exist. Only one layer can not bring nonlinearity in the network. One neuron network means it's a linear regression or logistic regression if it passes ...


1

ReLU and sigmoid have different properties (i.e. range), as you already noticed. I've never seen the ReLU being used as the activation function of the output layer (but some people may use it for some reason, e.g. regression tasks where the output needs to be positive). ReLU is usually used as the activation function of a hidden layer. However, in your case, ...


1

Just as a general remark, notice that technically we don't use the term "accuracy" for regression settings, such as yours - only for classification ones. If RMSE is 'in the units of the quantity being estimated', does this mean we can say: "The network is on average (1-SQRT(0.019))*100 = 86.2% accurate"? No. The advantage of the RMSE, as you have ...


Only top voted, non community-wiki answers of a minimum length are eligible