6
$\begingroup$

I'm a professional game developer investigating the potential for using reinforcement learning to build strategy game AI opponents that have more creative behavior compared to traditional techniques like behavior trees. I have a few questions I've bolded below, any thoughts would be helpful, and could save me from pursuing dead ends.

I created a very boring and tiny game as a test case. Two players each control a fleet of ships, each ship has health and can fire on one other ship each turn dealing some damage. The player and his opponent assigns orders to their respective ships, telling them which target to attack, and then the turn is resolved. Ships with 0 health are removed from the board. The player that loses all their ships first loses the game.

Assuming I was using TensorFlow, at a very high level I need to:

A) Create a training program that outputs a trained graph to a file. The training program will need to map gamestates into tensors, feed the tensors through the graph to produce actions, execute actions on the gamestate to generate a new state, and evaluate the reward function for the new state. Repeat a bunch.

B) Take the graph created in #1, load it at the game runtime, and use the graph to generate intelligent actions from real gamestates during the Player vs AI match.

As soon as I started digging into TensorFlow, questions immediately came up, and now I'm not quite sure if there is a more appropriate library to do this.

I) TensorFlow has a High Level python API, and a Low Level C++ API. Most games are built in C++, and thus using a C++ or C API is preferable, it makes integration with the game much simpler. In principle we could use pybind or some other scheme for sending state from C++ to Python and back again, but that's not ideal. Question 1: How much do I lose by using the low level API specifically for reinforcement learning, compared to the high level API?

II) Platforms. 99.9% of the time, PC/Console games are developed in Windows environments, and so having Tensorflow work in Windows is critical. From my googling, Tensorflow just barely supports building in Windows using CMake, though it requires some finagling. More worryingly are other platforms: Question 2: What hope is there of running the TensorFlow library on consoles like XboxOne, Playstation 4, or the Switch? I imagine this would require manually porting the entire source :(

III) TensorFlow is big, and it seems you need to basically link all of it to ship with the game. Question 3: Is there any way to get a slimmed down "Runtime TensorFlow" library that is only capable of loading a graph and transforming states into actions? It seems like if the answer was yes, it might also be easier to port this smaller runtime version to more platforms.

Question 4: Should I even be using TensorFlow for this? Is there perhaps something more suitable?

Thanks again if you read all that, I'm eager to start tinkering, but would like to set off in the right direction.

$\endgroup$
  • $\begingroup$ Welcome to SE:AI! My recent, brief forays into a few AAA strategy games indicate this type of initiative is still sorely needed! $\endgroup$ – DukeZhou Jun 14 '18 at 21:24
  • 1
    $\begingroup$ Thanks! I did have an offline convo with someone I know in the machine learning community, with some good advice: Caffe seems better suited to my use case, with the entire API in C++, and good support for deploying a CPU only mode. I'm going to implement a simple Q learning table for my trivial case before swapping in a neural net. The operations that a neural net performs to map an input into an output are also fairly simple, giving me hope that it may even be possible to take trained data from some machine learning package, and build a custom runtime that can run the net. $\endgroup$ – Stefan Lednicky Jun 16 '18 at 0:25
1
$\begingroup$

Investigating reinforcement as a way of producing more interesting behavior than behavior trees for AI based commercial strategy game development is a good idea. The simple test game given can be described briefly.

  • Human pitted against AI
  • Each player gets a fleet of ships
  • Each ship begins with a positive health level
  • One ship can be ordered to fire one shot at one ship per turn
  • Each hit produces a decrease in ship health
  • Ships with non-positive health are removed
  • Player with no remaining ships are removed
  • The remaining player wins

The question is correct in that game states must be represented as tensors. Ideally, they should be terse representations, with minimal redundancy. The actions must be represented as tensors too. The goal to train a network to play a game is generally feasible, but the simple game may not provide sufficient complexity to lead to learning. It is probably too simple. What is there to learn? In terms of the expectation regarding the result of the game, what advantage is there to firing on one ship over another?

It may be useful to understand that although machine learning can involve networks that can be represented as directed graphs, it is not the graphs of the network cells that are saved or loaded to persist learned behavior. The networks are parameterized, and those parameters are the representation of what is learned. Where this is most important is that, provided the form of the representation is standardized, the learning hardware and software environment need not match that of the environment that later leverages what is learned. The comment, "Build a custom runtime that can run the net [using the previously learned parameters]," is the correct method if this approach is taken.

In reinforcement learning, the representation is less like a graph and more like a structure involving a flexible behavioral policy that is affected by a changing distribution of probabilities. The expectations of future game states resulting from actions are part of the evaluation that must occur before each move. One theoretical approach to automated game play, for which a few varieties exist, is Q-learning. There are others using Markov trees and recurrent networks.

If the approach is to learn something in the lab that can be used in an xBox or Playstation, then supervised learning is indicated, and a high level language may assist in rapid prototyping. The result of training would be saved in language independent form, therefore readable by the portable c/c++. If the approach is to continuously change, then the reinforcement technology must be running on those platforms and most therefore be contained in statically or dynamically linked portable c/c++.

Both of these development approaches can be used in conjunction. For this basic investigative work and probably the first year or two of R&D, the use of Python for rapid prototyping would not require the use of Pybind. For the most common operations, the interoperability between Python and the mature C code ported from FORTRAN for tensor processing years ago is already present. The Python should never run on the game platform. It is not compiled and runs orders of magnitude less efficiently than C/C++ or Java. Determining whether the C++ API of tensorflow ports well to the desired target game environments is part of the architecture process. The Tensorflow underlying C++ API that Python ML newbies aren't generally concerned with is worth review by those who develop commercial games.

If 99.9% of the time, games are still developed on a Microsoft OS, game developers may be excessively patient. Fedora LINUX is much more productive for C, C++, Java, and Python development for both low level and high level programming tasks.

None of this has addressed the mention of a more creative opponent in the question.

The classic training goal in AI game play is to produce winning behavior, but games don't sell when the human always loses. Setting reward and loss functions to optimize on the basis of victory will not create sustainable sales of commercial game products. What a game developer might more wisely chose as a basis for optimization is engagement. Sacrifice victory in favor of sensation. You want gamers to blog to each other and talk in school and work about some crazy and unexpected thing the game did. Consequently, the goals of the learning must be adjusted in terms of training to maximize game value based on these product market realities.

$\endgroup$
0
$\begingroup$

I would recommend not to use TF at all, but use Pytorch instead. Make your test game in python. Even for simple game training can take a lot of time. If you are not afraid of some math and complexity you can start from AlphaZero General - it have structure which allow to plug any simple two-player deterministic game into it. That project is derivative of famous AlphaZero and on this moment is the most powerful RL method for those type of game. Be warned that even easy 6x6 game of Othello take tree days to train on single GPU with this method. After you get some experience with python prototype you can switch to C++ API.

Trying to run any framework on console though would probably be hellish and likely impossible. Most of frameworks use NVIDIA GPU tensor library "cudnn". I have no idea if you can access it from console. You can run framework on CPU and probably port it too, not using cudnn, but forward run (actual playing, not training) would be ~50 time more slow. If GPU run would be less 1 second then CPU AI would still be playable for turn-based game.

You can run any framework on linux and on windows and Mac (probably with some problems, but solvable). So for PC-games it's doable, assuming you manage to train python prototype.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.