What are the major differences between cost, loss, error, fitness, utility, objective, criterion functions?

Question

I find the terms cost, loss, error, fitness, utility, objective, criterion functions to be interchangeable, but any kind of minor difference explained is appreciated.

Answer 1

They are not all interchangeable. However, all these expressions are related to each other and to the concept of optimization. Some of them are synonymous, but keep in mind that these terms may not be used consistently in the literature.

In machine learning, a loss function is a function that computes the loss/error/cost, given a supervisory signal and the prediction of the model, although this expression might be used also in the context of unsupervised learning. The terms loss function, cost function or error function are often used interchangeably [1], [2], [3]. For example, you might prefer to use the expression error function if you are using the mean squared error (because it contains the term error), otherwise, you might just use any of the other two terms.

In genetic algorithms, the fitness function is any function that assesses the quality of an individual/solution [4], [5], [6], [7]. If you are solving a supervised learning problem with genetic algorithms, it can be a synonym for error function [8]. If you are solving a reinforcement learning problem with genetic algorithms, it can also be a synonym for reward function [9].

In mathematical optimization, the objective function is the function that you want to optimize, either minimize or maximize. It's called the objective function because the objective of the optimization problem is to optimize it. So, this term can refer to an error function, fitness function, or any other function that you want to optimize. [10] states that the objective function is a utility function (here).

A utility function is usually the opposite or negative of an error function, in the sense that it measures a positive aspect. So, you want to maximize the utility function, but you want to minimize the error function. This term is more common in economics, but, sometimes, it is also used in AI [11].

The term criterion function is not very common, at least, in machine learning. It could refer to the function that is used to stop an algorithm. For example, if you are executing a computationally expensive procedure, a stopping criterion might be time. So, in this case, your criterion function might return true after a certain number of seconds have passed. However, [1] uses it as a synonym for the objective function.