Do we need to know or verify properties of loss functions / metrics' implementations?

Question

I will start with an example, in order to get to the general question.

I was reading the following paper (https://www.cns.nyu.edu/pub/lcv/wang03-preprint.pdf) about Structural Similarity Index (SSIM), which is a function used in computer vision. Basically, given two images, it returns (according to some criteria) "how similar" these images are.

But what strikes me is that, in the paper, the following is stated: "we also would like the similarity measure to satisfy the following conditions".

I'll explain these properties now, but my question is the following: why we would like a function to satisfy some properties? In other words, what I understand is that it is interesting to prove that these properties hold, am I right?

Some years ago, I used SSIM not only as a metric to measure the performance of some algorithms, but also as a loss function itself. However, I definitively did not know about these properties, so it could be the case that I was optimizing (or measuring my results) with an implementation of the function that does not hold these properties, is that so?

As for the properties, these are straightforward:

• Unique maximum: S(x, y) = 1 if and only if x = y
• Boundedness: S(x, y) ≤ 1
• Symmetry: S(x, y) = S(y, x)

So the problem of "proving" SSIM's properties is useful for me for to raise the next (more general) research question: do AI-developers usually now properties of their loss functions/metrics? Are these properties relevant (e.g., are there functions for critical tasks)? Are they already being verified?

I would appreciate some insight about this.

Answer 1

Why we would like a function to satisfy some properties?

If we're talking about a loss function, you need to prove at the very least that the function has a minimum, otherwise you can't expect it to converge. Other properties like symmetry and boundedness are not strictly required but they can help avoiding exploding loss scores as well as improving convergence time and convergence on corrupted labels.

If we're talking instead of a metric or similarity score then properties like having a unique maximum become compulsory. In the case of SSIM for example you want the function to return the maximum score only when comparing an image to itself. Symmetry is also a pretty obvious requirement, since comparing A to B should lead to the same score as comparing B to A.

Do AI-developers usually know properties of their loss functions/metrics? Are these properties relevant (e.g., are there functions for critical tasks)? Are they already being verified?

From the paragraph above it should be clear that yes, these properties are relevant (not always essentials). A good developer should know which properties a function respect, many times it is as simple as taking a look at the function graph, otherwise referring to the original paper is always the safest option, also if you want to check if the authors bothered to mathematically prove that the properties always hold (which they almost always do if they want to get published).