Assuming we had an unlimited time to train a model and a very powerful machine to use our model in real-time (hello quantum computer), I'd like to know why no one could achieve to build an AI able to play a FPS, using ONLY pixels shown on the screen.
Disclaimer: I am not tackling this problem and neither am I planning on doing such a thing, this is pure speculation and curiosity.
I read this great article: playing FPS Games with Deep Reinforcement Learning (2017) (Guillaume Lample, Devendra Singh Chaplot) where they achieve a 4/1 kills/death ratio on Doom against bots. But this is 3 years old now.
Here is a picture of their model:
But they made 2 assumptions that we, humans, do not make when we are playing for the first time to a new game like Call of Duty or Battlefield:
Game feature augmentation. To train a part of their model they used the game engine to know if there is (or not) an enemy in the frame they are processing. We obviously can't do this with CoD or Battlefield, and we, as human, just "learn" to recognize an enemy without these informations.
Changing textures of several maps while training to generalize better the model (see 5.2 of the paper linked previously). To summarize, they trained the model with 10 maps changing texture of some elements to make the model generalize better. Then they tested it on 3 unknown maps. In real world (ie in the scenario where we base our training/testing exclusively on the pixels of the screen), we can't train a model with different textures on the same map. And humans are able to play a deathmatch on an unknow map without re-learning everything (detect enemies, move, hide, reloading...). We just need to construct a 3D model of the map in our head to play our best.
Their agent "divides the problem into two phases: navigation (exploring the map to collect items and find enemies) and action (fighting enemies when they are observed), and uses separate networks for each phase of the game".
Would it be wise to use more than 2 models? Let's say:
- 1 CNN to detect enemies
- 1 model to deal with space features (position/navigation of the agent and the enemies)
- 1 model to choose the actions given all data previous models have found?
We could train them independently, at the same time.
I think we'd get better result by processing manually some features (using computer vision techniques) like the minimap to get know positions of enemies and number of ammo to feed as input of the last model (action decider).
But there are other problem we'd get: there is a delay between the frame we choose to pull the trigger, the time the bullet hit the enemy and the time the "reward" appears on the screen (ex: "100 points, kill [nameOfLateEnemy]" appears after the 3th bullet, and if there is ping because we are playing online it may appear 100ms after). How to use reinforcement learning when we don't know exactly which action was the one getting the reward? (we can move the agent while changing the lok directon while pulling the trigger at the same time. It's the combination of these actions that is making the agent kill an enemy).
If the 2 assumptions they made were easy to get rid of, they would be discarded already.
However detecting enemies is basically a simple CNN, and making the navigation network generalize certainly have solutions I can't think of right now but some researchers should have in this 3-year gap between the paper and today.