0
$\begingroup$

We are trying to evaluate a loss function on the Year Prediction (Million Songs) data set. The problem is that we don't know how to configure an experiment in order to test if one loss (the standard Mean Squared Error, MSE) or a quadratic one is better, on a 10-fold cross validation model validation framework.

At present we are using a net made of 10 fully connected layers of 100 nodes each, but we don't know if this is the best architecture, since we are getting training errors not below 80.0 (in terms of MSE). That is, we're unable to fit the data. We've tried other nets (with added layers) but it didn't help.

You can find the python code and the data set in:

https://osf.io/2zhyc/

$\endgroup$

1 Answer 1

0
$\begingroup$

The way to evaluate any supervised learning result is to pick a metric - a scoring system for the results. Ideally this metric captures key details of what properties you care about for the trained model. For instance in your task you could arbitrarily have a metric of +1 for getting the year correct, or +0.25 for getting decade correct if the year is not corrrect.

There is no fixed rule about which metric function is better. If you can translate successes and failures of the prediction system into comparable benefits and costs (e.g. financial rewards and losses for getting things right or wrong), then that would be a good choice, because your metric on the test dataset can be a prediction of financial reward for deploying the trained model. With most AI projects though, this is not really feasible, and you will need to create some proxy measure that captures the goals of the project.

You can use loss functions directly as metrics (it is common to use MSE as a metric for example in regression problems). If you select a loss function for this purpose, it is likely that your best architecture will also use the same loss function as doing so will directly minimise the target metric in the learning process.

If you want a fair comparison of results using different loss functions, in order to find a "best" one, it is important to select your metric function first, before running experiments.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .