I came across 'Amber'(https://ambervideo.co/) where they are claiming that they have trained their AI to find patterns emerging due to artificially created videos which are invisible to naked eye.

I am wondering that the people who are creating deepfakes can as well their AI's to remove these imperfections and so the problem reduces to 'cat-mouse' game where having more resources(to train their AI) is more crucial.

I do not work in AI and vision and so I may be missing some trivial points in the area. I would really appreciate if detailed explanation or relevant resources are given.

Edit: Most of the people who do manipulate the media news or create fake news could afford more resources than an average citizen. So, is the future is really going to be dark where only few strong have even more control on the society than today?

I mean even though there are fake photos created by photo shop, most of the good photo-shopped photos do take a long time to make. But if AIs can be trained to do that then it is more about having large resources. Are there related works which give hope to know real from fakes?

P.S.: I realize that after the edit, the question also went tangential to the topic-tags here. Please let me if there are relevant tags.

  • $\begingroup$ Yes, it will become a cat and mouse game $\endgroup$
    – mshlis
    Commented Aug 12, 2019 at 13:29
  • 1
    $\begingroup$ No it won't eventually the generator will generate perfect images and the detective AI if classify it as false image will also classify real images as false images. Basically we will probably reach advanced cryptographic algorithm like stages where the probablity of detection is so low that it's not possible by any amount of resources. (Unless we voluntarily introduce some noise) $\endgroup$
    – user9947
    Commented Aug 12, 2019 at 13:44

3 Answers 3


I think this game will go pretty crazy, because, at some point, the generator AI will be able to generate absolutely perfect images. Actually, no, just perfect enough that no AI can be sure whether they are real or fake.

So, I think the AI war will go onto more than the image, the detector AI will probably evolve to analyze whether this video is logically plausible, for example, by tracking the celebrities' position to prove that it is impossible that he/she was, for example, let's put it this way, being unloyal to his/her partner.

I mean, currently, AI can tell whether an image is fake or not better than human because it has seen about a million times more samples than us, but if we know who the person in the image is and we are as stalky as the AI I just described, we can probably work out that this image is implausible.

Of course, there will be counter measurements to that. But, at that point, we might as well just let the AI rule the world, given that it will have become this smart (lol).

But, seriously, if it's smart enough to think this far ahead in this 'real world' problem, then strong AI is nigh.


As mshlis begins to touch on, yes we can. However, it will be an unending war. There are quite a few reasons for this. For one, the problem itself is not simple. There are many different 'versions' of the deepfakes framework out in the wild at this point, any algorithm you create to try and spot them would have to work for all of the different iterations. Another reason is the systems that would be used to combat it can be quite easily fooled(see).

However, the most glaring, and unending problem comes from the architecture itself. Let us say we create a perfect algorithm that is foolproof and extremely accurate. Even then, all one would have to do is use that algorithm as the discriminator during training of your deepfakes model, and bing-bang-boom, your deepfake detection model is busted.

  • $\begingroup$ So the architecture is invertible? $\endgroup$
    – user9947
    Commented Aug 12, 2019 at 18:35
  • $\begingroup$ @DuttaA in this case we would be using the “perfect” discriminator, or the one that can tell it’s a deepfake, as the discriminator to train the new generator. Apologies if that was unclear $\endgroup$ Commented Aug 13, 2019 at 4:21
  • $\begingroup$ You said 'foolproof and extremely accurate'. Is it accurate in generation or detection? $\endgroup$
    – user9947
    Commented Aug 13, 2019 at 4:32
  • $\begingroup$ @DuttaA, yea sorry again, I am rambling there wrt an algorithm that would accurately predict whether a given input was a deepfake. We then would plug that into our architecture as the discriminator for our deepfakes generation. Edited it to hopefully make it a bit more clear(some things sound simpler in one's head) $\endgroup$ Commented Aug 13, 2019 at 5:09

Sound and image manipulation necessarily creates artifacts. Around the edges of superimposition in layers there are such. Face replacement and other more surface or object centered operations create a different class of artifacts. A sufficiently well constructed LSTM or GRU network and data set of manipulated frame sequences and the user (mouse and keyboard) events that manipulated them can be used to produce good guesses of the event set from new images. Adding unmanipulated images to the data set can allow for the no-event case. That would be the supervised way to do it. There are unsupervised approaches that would require considerably less training resources, which is likely the case with this San Francisco solutions provider.

In either case, the question of escalation is a good one. One can also create a device, building from the current state of machine learning, that hides manipulations from existing detection software. If they are forward thinking, the same provider may have already developed it.

Can we combat against deepfakes? ... I am wondering that the people who are creating deepfakes can as well their AI's to remove these imperfections ...

Yes and yes. In war, the combatants learn the methods of the opposing combatants and adapt. A detection mechanism for opposing strategy changes is also theoretically possible, which is one of the reasons that military research facilities spend so much on higher forms of AI.

The edit to the question is not entirely tangential either.

If we propose, which some people have, that a virtual reality may damage human culture or individual psyches, the average citizen is likely to be considered collateral damage on the field of combat by companies seeking a good financial return from their AI development. Of course, we could say the same thing about the use of diminished fifths in music. Two notes that are six half steps apart produce a dissonant frequency ratio of $1:\sqrt{2}$. The diminished fifth was considered subliminally satanic in Europe centuries ago and prohibited in music compositions by law. The glass harmonica was alleged to have driven listeners insane.

Anthropologically, it is possible that a mark of our species is to manipulate appearance. To hunt fakes in frames and audio is likely a fruitless hunting ground, with our without the escalation. The current hunting ground of import is the research into what genetic elements led to human abilities to imagine, design, and fabricate. After that is known, we may have a better window into whether the cat-and-mouse games we play have any sustainable value for our species going forward. Those who love competition believe that it strengthens, which is possible. It is also possible that the games are solely an artifact of a painful path to our emergence as the dominant mammalian species and no longer of any particular use. "Do to others as you would want them to do to you," has the ring of truth we can't ignore either.

If we look through this wider lens, we can see that our entertainment choices tend toward what could (in the absence of bias) qualify as deepfakes. There are entire cities fueled by the money made by the entertainment industry producing excellence in sound and image capture, synthesis, and manipulation. The story lines are not necessarily representing deep truths. This is more overt.

On the more covert side, some pass fakes off as reality as a move in their own game to achieve some objective, but this is not the exception in our culture. The fields of public relations and marketing are based on the creation and preservation of business value. Some elements of government, education, and community are based on the creation of economy-preserving beliefs. The intention may be to benefit others or beat them and gain personal wealth.

Some of us seek authenticity and would like the fake-finders to win the combat, but it appears they may be on the losing side.

Does this question and this answer pertain to this Stack Exchange community? Absolutely. This community's description in the drop down of SE communities reads, "For people interested in conceptual questions about life and challenges in a world where 'cognitive' functions can be mimicked in purely digital environment." Whether AI ultimately weighs in on the side of playing people or informing them certainly pertains to this published view of this community's purpose.


Not the answer you're looking for? Browse other questions tagged .