# How should I design the action space of an agent that needs to choose a 2d point and then shoot a cannonball?

I'm building a game environment (see the picture below) where an agent should position the mouse on the screen (see the coordinates on the upper right corner) and then click to shoot a cannonball. If the goal (left) is hit. The agent gets a reward based on the elapsed time between this strike and the last one. If three shots are missed, the game is done and the environment will reset.

The env is done so far. But now I wonder what the action space should look like. How can I make the agent choose some x and y coordinates? And how can I combine this with a "shoot" action?

• you can learn the mean $\mu$ of normal distribution for $x$ coordinate position and for $y$ coordinate position. Then the action would be sampled coordinates from the normal distribution. You can also learn standard deviation $\sigma$ or it can be fixed. You don't need to model shoot action, agent can shoot immediately to given coordinates. You would need to use policy gradient/actor critic method for this Commented Mar 10, 2020 at 11:07

You could make the actions

1. move up
2. move down
3. move left
4. move right
5. shoot

Then you can declare the speed of this movement. If you want to go more in depth you can do

1. Accelerate up
2. Accelerate down
3. Accelerate left
4. Accelerate right
5. Shoot

This will give control of the speed to the neural network, but is harder to train, and you should give the speed of the mouse as an input.

• I also thought about that. But since the score is based on the time between two hits I see some problems. Because the main goal is that the agent gets as fast as possible. But when can can only move x much pixels it will lack precision and speed I think...
– Voß
Commented Mar 10, 2020 at 10:43
• I don't think it will lack precision. And if you use the acceleration the speed purely depends on how much you accelerate it on every step, and how many frames you wait to make the next step Commented Mar 10, 2020 at 11:14