It this podcast between Oriol Vinyals and Lex Friedman: https://youtu.be/Kedt2or9xlo?t=1769, at 29:29, Oriol Vinyals refers to a paper:

If you look at research in computer vision where it makes a lot of sense to treat images as two dimensional arrays... There is actually a very nice paper from Facebook. I forgot who the authors are but I think [it's] part of Kaiming He's group. And what they do is they take an image, which is a 2D signal, and they actually take pixel by pixel, and scramble the image, as if it was a just a list of pixels, crucially they encode the position of the pixels with the XY coordinates. And this is a new architecture which we incidentally also use in Starcraft 2 called the transformer, which is a very popular paper from last year which yielded very nice results in machine translation.

Do you know which paper he is referring to?

I'm guessing maybe he is talking about non-local neural networks, but I'm probably guessing wrong.

Edit: after reviewing the recent publications of Kaiming He (http://kaiminghe.com/), maybe I'm guessing right. Any thoughts?

  • $\begingroup$ If you verify please do update. I’d like to read that, too. Also, does this podcast have show notes? Sometimes they include links to referenced papers / blogs, etc. $\endgroup$ – Hanzy May 1 '19 at 16:47
  • 2
    $\begingroup$ This isnt the paper but it sounds related. "[An intriguing failing of convolutional neural networks and the CoordConv solution][1]" from Uber AI Labs. They also have a [really informative video][2] and a [repo][3]. [1]: arxiv.org/pdf/1807.03247.pdf [2]: youtube.com/watch?v=8yFQc6elePA [3]: github.com/uber-research/CoordConv $\endgroup$ – Jaden Travnik May 1 '19 at 17:18

So this paper is by google, but is very similar where they use 2D positional embeddings and perform MHA on the flattened image. Are you talking about Attention Augmented Convolutional Networks

| improve this answer | |
  • $\begingroup$ Looks like this is not "the" paper, but it's a good find, thanks $\endgroup$ – Benjamin Crouzier May 13 '19 at 19:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.