1
$\begingroup$

Consider the following abstract from the research paper titled A Note on the Inception Score for instance

Deep generative models are powerful tools that have produced impressive results in recent years. These advances have been for the most part empirically driven, making it essential that we use high-quality evaluation metrics. In this paper, we provide new insights into the Inception Score, a recently proposed and widely used evaluation metric for generative models, and demonstrate that it fails to provide useful guidance when comparing models. We discuss both suboptimalities of the metric itself and issues with its application. Finally, we call for researchers to be more systematic and careful when evaluating and comparing generative models, as the advancement of the field depends upon it.

Here we can observe the usage of word metric several times. In mathematics, the word metric is used only in the context of metric spaces afaik. The definition for metric space and metric is defined as follows

A metric space is a set $X$ together with a function $d$ (called a metric or "distance function") which assigns a real number $d(x, y)$ to every $x, y, z$ belongs $X$ satisfying the properties (or axioms):

  1. $d(x, y) \ge 0$ and $d(x, y) = 0$ iff $x = y$,
  2. $d(x, y) = d(y, x)$,
  3. $d(x, y) + d(y, z) \ge d(x, z).$

Do research papers generally use the word metric in the sense of the metric defined above? Or do we need to interpret the word metric less rigorously, just as a measure, like an accuracy?

Note: Although I provided the abstract from a research paper containing the word metric, the question is not restricted to this particular context. This question can be applied to all AI-related research papers that used the word metric, especially in the context of performance or evaluation metrics.

$\endgroup$
2
  • 2
    $\begingroup$ I don't think you should interpret it rigorously. Most of the "metrics" used turn out to satisfy those constraints, but I think I've seen KL divergence-based "metrics" being called, well, metrics, while they do not satisfy property 2 $\endgroup$
    – Ant
    Apr 5, 2022 at 20:30
  • 1
    $\begingroup$ @Ant That looks like the beginning of an answer. In addition to KL divergence, maybe you can provide more examples of "metrics" that are or not really metrics by showing that they satisfy or not the properties of a mathematical metric. $\endgroup$
    – nbro
    Apr 6, 2022 at 14:01

1 Answer 1

1
$\begingroup$

"Metric" should be understood as "a function of the trained model and of a dataset which returns a number".

For example, in reinforcement learning, one can use as an evaluation metric the total cumulative discounted reward that a policy gets on a set of episodes for a given task, starting from the same initial state $S_0$.

That's a function $f(\tau_1, \tau_2, ... | \pi, S_0)$, where $\tau_1$ is the first trajectory obtained by running policy $\pi$ on the task, $\tau_2$ the second one, etc. (we assume that $\pi$ is stochastic, otherwise it doesn't make sense to run multiple episodes). $f$ is an "evaluation metric" but is certainly not a distance function.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .