How to approach a two-agent two-step action game?

Question

A simple two-player sniper game:

• Each player has 9 houses that he can reside in. So 18 houses in total. The houses can be considered in a row: e.g. 1-9 for player A, and 10-18 for player B.

• Each step, the player should make two actions! First, he can use his gun's limited view to check out three consecutive houses of the enemy to see if he is there (for example, he choose 3,4,5.). Then, based on that result, he can choose one house to shoot. That means if he guessed correctly, he will know the other player is in one of those three houses. Otherwise, he can shoot one of the remaining six houses.

• The killer wins!

---

Please note that in each step, the player has to perform two actions without interruption from the other player. Based on the result of the first action (limited view), he will have more information to select his second action (shooting). Thus, the first action is informative to reduce action space.

---

I have decided to use stable-baselines3. I have to create an environment. I am not sure about the policy network.

How should I approach this game for training an AI agent? I would really appreciate it if you can guide me on env creation, policy selection, or any general tips.

Answer 1

I am not sure how in-depth the information you want need to be, but maybe I can share some thoughts that may help you!

Environment I would highly recommend using OpenAI Gym with your environment, since most of the already implemented RL-algorithms are designed to use gym environments. You can design your environment any way you want to and then use gym to exchange action, observation and reward handling. If you want a 3D environment you could look into Unity3D which has a machine learning toolbox, but as far as I understood you can just model the game in python.

Algorithm Now the tricky part. It really depends on how you design your game and how you want to handle the agents. A simple scenario would be only using one agent (Single-Agent RL). In this case, you would have to code a simple opponent by yourself and the agent could only be as good as the opponent you coded.

If you want them both to be as good as possible, you would have to use a Multi-Agent RL (MARL) approach. MARL comes with its own problems and difficulties. The main problem is that the learning environment is unstable, because the resulting states and rewards do not only depend on the action of one agent, but of the actions of multiple agents. In a competetive game, where the agents do not communicate, Independent Reinforcement Learning (IRL) could work. In this case you would use two completely seperated agents with a single-agent RL algorithm for each agent.

However other MARL algorithms make use of centralised training, which guarantees a stable learning environment and decentralised execution. MADDPG is a wildley used algorithm for MARL problems, however there are a lot of different and newer algorithms. I would recommend checking recent review papers like this one.

How I would approach the problem Now I don't know how helpful this was. If I had to train agents for your scenario, I would follow these steps:

1. Create the environment and test it
2. Design an observation space - what does the agent see in each time step?
3. Decide on action spaces (discrete in your case), which action can he choose?
4. Design a reward function / reward to tell the agent when the action was successful
5. Decide on an algorithm and train the agents with scenarios as simple as possible, then make it more difficult if the training is working.

Hope this helps!