Why do adversarial attack transfer well?

Question

I have read (*) that a common technique to attack a black box AI system based on a neural network is to use it to train a surrogate model to make the same classifications as the black box one.

Once this is done, one can look for adversarial examples on the surrogate model (on which the attacker has access to all the weights and can compute gradients).

The key property that makes such attacks successful is transferability: an adversarial example on the surrogate model is likely to be an adversarial example on the black box model.

Question: Do we know why such transferability properties hold and under which conditions?

(*) Although I don't remember where I first read it, this is mentioned in the book Not with a bug but with a sticker although it doesn't get into technical details (this is a book for the general public).

Answer 1

As far as I know, there are only hypothesis on this (https://simons.berkeley.edu/talks/new-perspective-adversarial-perturbations) which pretty much relies on the idea that there are "spurious" correlation in the dataset used for training (check the video in the link for more informations, they have also some papers referenced to read more about it)