I'm a professional game developer investigating the potential for using reinforcement learning to build strategy game AI opponents that have more creative behavior compared to traditional techniques like behavior trees. I have a few questions I've bolded below, any thoughts would be helpful, and could save me from pursuing dead ends.
I created a very boring and tiny game as a test case. Two players each control a fleet of ships, each ship has health and can fire on one other ship each turn dealing some damage. The player and his opponent assigns orders to their respective ships, telling them which target to attack, and then the turn is resolved. Ships with 0 health are removed from the board. The player that loses all their ships first loses the game.
Assuming I was using TensorFlow, at a very high level I need to:
A) Create a training program that outputs a trained graph to a file. The training program will need to map gamestates into tensors, feed the tensors through the graph to produce actions, execute actions on the gamestate to generate a new state, and evaluate the reward function for the new state. Repeat a bunch.
B) Take the graph created in #1, load it at the game runtime, and use the graph to generate intelligent actions from real gamestates during the Player vs AI match.
As soon as I started digging into TensorFlow, questions immediately came up, and now I'm not quite sure if there is a more appropriate library to do this.
I) TensorFlow has a High Level python API, and a Low Level C++ API. Most games are built in C++, and thus using a C++ or C API is preferable, it makes integration with the game much simpler. In principle we could use pybind or some other scheme for sending state from C++ to Python and back again, but that's not ideal. Question 1: How much do I lose by using the low level API specifically for reinforcement learning, compared to the high level API?
II) Platforms. 99.9% of the time, PC/Console games are developed in Windows environments, and so having Tensorflow work in Windows is critical. From my googling, Tensorflow just barely supports building in Windows using CMake, though it requires some finagling. More worryingly are other platforms: Question 2: What hope is there of running the TensorFlow library on consoles like XboxOne, Playstation 4, or the Switch? I imagine this would require manually porting the entire source :(
III) TensorFlow is big, and it seems you need to basically link all of it to ship with the game. Question 3: Is there any way to get a slimmed down "Runtime TensorFlow" library that is only capable of loading a graph and transforming states into actions? It seems like if the answer was yes, it might also be easier to port this smaller runtime version to more platforms.
Question 4: Should I even be using TensorFlow for this? Is there perhaps something more suitable?
Thanks again if you read all that, I'm eager to start tinkering, but would like to set off in the right direction.