1
$\begingroup$

I will start with an example, in order to get to the general question.

I was reading the following paper (https://www.cns.nyu.edu/pub/lcv/wang03-preprint.pdf) about Structural Similarity Index (SSIM), which is a function used in computer vision. Basically, given two images, it returns (according to some criteria) "how similar" these images are.

But what strikes me is that, in the paper, the following is stated: "we also would like the similarity measure to satisfy the following conditions".

I'll explain these properties now, but my question is the following: why we would like a function to satisfy some properties? In other words, what I understand is that it is interesting to prove that these properties hold, am I right?

Some years ago, I used SSIM not only as a metric to measure the performance of some algorithms, but also as a loss function itself. However, I definitively did not know about these properties, so it could be the case that I was optimizing (or measuring my results) with an implementation of the function that does not hold these properties, is that so?

As for the properties, these are straightforward:

  • Unique maximum: S(x, y) = 1 if and only if x = y
  • Boundedness: S(x, y) ≤ 1
  • Symmetry: S(x, y) = S(y, x)

So the problem of "proving" SSIM's properties is useful for me for to raise the next (more general) research question: do AI-developers usually now properties of their loss functions/metrics? Are these properties relevant (e.g., are there functions for critical tasks)? Are they already being verified?

I would appreciate some insight about this.

$\endgroup$
2
  • $\begingroup$ If the function doesn’t measure anything of interest, what’s the point? Why not just output $7$ every time? $\endgroup$
    – Dave
    Sep 19, 2022 at 16:32
  • $\begingroup$ I mean, the function has its own shape (with several calculi on covariances etc), but they also mention those three properties over the specified function. This is what I wonder: why specify some properties over an already-described function? How relevant are them? Are the cases out there where these kind of properties not holding yield wrong optimizations/measurements? $\endgroup$
    – Theo Deep
    Sep 19, 2022 at 16:36

1 Answer 1

2
$\begingroup$

Why we would like a function to satisfy some properties?

If we're talking about a loss function, you need to prove at the very least that the function has a minimum, otherwise you can't expect it to converge. Other properties like symmetry and boundedness are not strictly required but they can help avoiding exploding loss scores as well as improving convergence time and convergence on corrupted labels.

If we're talking instead of a metric or similarity score then properties like having a unique maximum become compulsory. In the case of SSIM for example you want the function to return the maximum score only when comparing an image to itself. Symmetry is also a pretty obvious requirement, since comparing A to B should lead to the same score as comparing B to A.

Do AI-developers usually know properties of their loss functions/metrics? Are these properties relevant (e.g., are there functions for critical tasks)? Are they already being verified?

From the paragraph above it should be clear that yes, these properties are relevant (not always essentials). A good developer should know which properties a function respect, many times it is as simple as taking a look at the function graph, otherwise referring to the original paper is always the safest option, also if you want to check if the authors bothered to mathematically prove that the properties always hold (which they almost always do if they want to get published).

$\endgroup$
2
  • $\begingroup$ Thanks a lot for the answer, it really clarifies my doubts. My only remaining question is the following: how do developers usually verify that the implementation of the function holds its properties (i.e., that it is implemented according to original paper's specifications)? $\endgroup$
    – Theo Deep
    Sep 20, 2022 at 9:49
  • 2
    $\begingroup$ Unfortunately the only way is to go trough the code and check that it respect the math described in the paper. If a repo is well made tough, the code should include scripts to reproduce the paper results, which is enough of a proof that the implementation does what it's supposed to. $\endgroup$ Sep 20, 2022 at 11:32

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .