How do I create an AI controller, which can play pacman - by taking in pixel values (or some other data by represents the state) which perhaps runs on a separate thread, which can control the game?

It takes the pixel values (or likewise) to learn, and play the game, just like a human does. Normally, AIs like enemies - are part of the game code, which learn and play by using the internal state of the game, just like the “enemy” AI.

How can I create this abstraction between the game and the agent?

The most obvious example would be : (I don’t know how viable this is) - The agent takes in pixel values of the game - learns - and sends “keyboard” inputs to that thread just like a human would, to play the game.

Can someone point me in the right direction - as to how to do something like this?

  • $\begingroup$ I don't understand this: "where the “agent” which learns is part of the game code itself. Meaning it’s just like any other AI in the game, but it learns dynamically.". What do you mean by 1. "just like any other AI in the game" (i.e. which "any other AI in the game" are you referring to? Are you referring to rule-based bots?), 2. "is part of the game code itself"? Do you mean that there are no abstractions that separate the "game" from the "agent"? If that's the case, how is that not just a programming issue (i.e. some bad programmer that doesn't know what abstractions are)? $\endgroup$
    – nbro
    Commented Jun 21, 2022 at 18:46
  • $\begingroup$ From "separated “controller” thread", it seems that you were talking about creating multiple threads (one for the agent and one for the game, whatever that means). If that's the case, rewrite your question to make that initial paragraph clearer. Maybe also provide the link to the code you were referring to. Which models or algorithms are you specifically referring to? Also, the title doesn't really match the problem or question in the title. One question per post, so the question in the title should be equal to the equation in the body. Edit your post to fix all these issues! $\endgroup$
    – nbro
    Commented Jun 21, 2022 at 18:47
  • $\begingroup$ The answer below just seems a general answer to "how can I create a game AI agent". It doesn't even focus on pacman, threads, or whatever you're really asking here. It assumes that you want to learn from pixels, while you write "taking in pixel values or something else.". So, this post and the answer below couldn't be more problematic, and I don't have an idea of why people upvoted this! $\endgroup$
    – nbro
    Commented Jun 21, 2022 at 18:53
  • $\begingroup$ Valid answer to a valid question. He just wants to know about deep rl (as he looks for something that dynamically plays games from pixel values (or other datatypes) and chooses actions like a player would (no tricks)), and not classical rl that is deeply integrated with the game (but learns dynamically). What is almost like simple rule based AI you generally find in games. $\endgroup$
    – hal9000
    Commented Jun 21, 2022 at 19:57
  • $\begingroup$ Your right the question is quite confusing. $\endgroup$
    – hal9000
    Commented Jun 22, 2022 at 8:09

1 Answer 1


About the environments
For the controller part of your question, I would advice looking at openAI gym. https://www.gymlibrary.ml/content/environment_creation/ #how to make your own gym enviroment
Those gym enviroments work kinda like this

env = gym.make("ALE/SpaceInvaders-v5")
observation = env.reset()

observation, reward, done =env.step(action) 

Where your observation can be pixels,ram dumps, etc. The actions can be internally mapped to key presses. (but the internals don't really matter that much) Gym environments are just an easy way to then have your agent take in the observations and map them to actions. So your game can be stepped through so you don't have to worry about a complicated way to integrate the AI and keeping it synced.
There are already a bunch of artari games included (also ms pacman) https://www.gymlibrary.ml/environments/atari/

About the AI

Since you want to learn directly from the pixels or other more complex data types I advice you to read this:

Deep Reinforcement Learning, a textbook (arXiv:2201.02135) https://arxiv.org/abs/2201.02135

If you just want some simple tutorial just read "Hands-On Reinforcement Learning with Python", then you can try to implement something like DQN , with some cnn architecture. (similar to how they did in arXiv:1312.5602 "Playing Atari with Deep Reinforcement Learning"). Since DQN is model free, off policy and relatively easy to implement. (cool thing about off policy is that your agent doesnt have to interact with the environment as it doesnt need to sample "experience" using only it's current policy (way of choosing actions given a state) so you can even play the game itself and collect information (a,s,s',r) while playing and train your agent on that (to nudge it in the right direction if it gets stuck))

if you're not a fan of gradient descent based methods, you can also use methods like ES (evolutionary strategies). As that way you can directly optimize for a reward. (it's a genetic algorithm that scales well in both compute and dimensionality) (arXiv:1703.03864 Evolution Strategies as a Scalable Alternative to Reinforcement Learning)

  • $\begingroup$ (a,s,s',r) stands for action, state, next state, reward. $\endgroup$
    – hal9000
    Commented Jun 21, 2022 at 13:45
  • $\begingroup$ I "advice" you to ask for clarifications before providing an answer because the question and post are currently unclear. $\endgroup$
    – nbro
    Commented Jun 21, 2022 at 18:57
  • $\begingroup$ It's quite clear what he wants, he is just very unfamiliar with rl. He is dissatisfied with the idea of simple AI that follows some simple rules and wants some separate agent that looks at pixels (like the player) and also can only do the same actions as the player (while poorly formulated it's quite clear). So seeing that I point him into the right direction being deep rl textbooks (since he wants the AI to learn pacman from the pixels). $\endgroup$
    – hal9000
    Commented Jun 21, 2022 at 19:14
  • $\begingroup$ The OP writes "taking in pixel values or something else.", so why are you so sure that the OP wants something that takes (only) pixel values? The OP may be unfamiliar with RL, but that doesn't mean that the question is clear. The title says "how to create an AI controller for pacman", which is different from "create something like a separated “controller” thread". The OP has already looked into implementations of RL for pacman, so I assume he's not so unfamiliar with it, otherwise, they would not have understood anything in those implementations. In my view, the post is just unclear. $\endgroup$
    – nbro
    Commented Jun 21, 2022 at 19:20
  • $\begingroup$ I don't give an answer that takes only pixel values. I point him to deep rl with that he has his controller that can interact with openAI gym to play his pacman directly from pixel values or ram states or whatever he wants. $\endgroup$
    – hal9000
    Commented Jun 21, 2022 at 19:24

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .