I find the terms cost, loss, error, fitness, utility, objective, criterion functions to be interchangeable, but any kind of minor difference explained is appreciated.
What are the major differences between cost, loss, error, fitness, utility, objective, criterion functions?
We can first consider what kind of elements are in this list. They are all quantifications that guide a process of adaptation, learning, or innovation — controls a process that optimizes or attempts to optimize the development of another process. Quantification allows for comparison and the determination of first and second-degree derivatives to use in approximating steps in improvement.
On the basis of this generalization, the list may be expanded to include other potential quantifications that can control.
In English, there is quite a bit of nuance in terms of normal contextual use. In academic papers on machine learning topics there are some conventions forming there, producing jargon, meanings beyond the normal English use. We can first divide them symmetrically around the most basic qualifier. Is the increase in the number what is intended or something to be avoided?
Decreases are Desirable
Increases are Desirable
Some of these are relative, such as Advantage being a relative measure between two Values associated with two Action options. Some are direct inversions, as in that the quotient of Gain over Loss equals -1 in the usual contexts of those words. Some appear to be opposite but in fact are not, as in the case of pain and pleasure occurring concurrently and via different neural mechanisms in biological neural systems.
Some are discrete and have only two possible values, such as Criterion and Complete. Those can be well represented by Boolean values rather than integers, fixed-point, or floating-point numbers.
Some are conditionally synonymous, as in the case of Distance and Error if the output dimensionality is one or the output is multidimensional but the error aggregation method used is the square root of the sum of squares.
Some seem better in financial contexts ... others in robotic contexts.
Words like Disparity and Optimality are generalizations.
One, in particular, is a quotients based on investment &mdash Return — which could be applied to energy return on energy invested (as Hall suggests is the fitness function for genetic evolution).
What is missing from this list is learning based on multiple degrees of freedom, not used in mainstream ML yet, and probably will emerge in papers and then libraries and then VLSI designs in time. This likely direction would introduce the plurals of many of these quantifications because it will be vectors of corrective signaling back-propagating to perform curved modifications on network parameters. It is likely that brain signaling (pulses, energy balances, neurochemical secretions and uptake, and genetic expression) will not be well simulated until this happens.
In all cases, it is the definition of these quantifications that drive adaptation, learning, and innovation in biological and artificial systems. It is thus likely that the application of ethics and risk analysis might best be applied to AI in the formulation of the calculation of these quantities in adaptive systems.
They are not all interchangeable. For example, in genetic algorithms, the fitness function does not usually involve an external signal (the labels), while (at least, in machine learning) a cost (or loss) function usually requires the external signal to compute the cost (or loss). However, all these expressions are related to each other and to the concept of optimization.
The objective function is the most general term that can be used to refer to a cost (or loss) functions, to a utility function or to a fitness function, so, depending on the problem, you either want to minimize or maximize the objective function. The term objective is a synonym for goal.
The expressions loss function, cost function or error function are often used interchangeably (at least, in machine learning). More precisely, you might prefer to use the expression error function if you are using the mean squared error (because it contains the term error), otherwise, you might just use any of these expressions interchangeably.
A utility function is usually the opposite or negative of a cost, loss or error functions, in the sense that it measures a positive aspect. So, you want to maximize the utility function, but you want to minimize the cost, loss or error functions.
The expression criterion function is not very common, at least, in machine learning. This expression might refer to the function that is used to stop an algorithm. For example, if you are executing a computationally expensive procedure, a stopping criterion might be time. So, in this case, your criterion function might return true after a certain number of seconds have passed.
There are other expressions related to the mentioned ones. For example, there is a reward function, which is used in reinforcement learning (RL) to refer to the function that determines the reward that is given to the RL agent when certain actions are taken from certain states.
To conclude, all these expressions are related, some of them are interchangeable and others have a similar meaning, but, depending on the context, field or problem, one is preferred over the other.