Training a reinforcement learning agent that can decide to continue or end the game

Question

I am trying to use reinforcement learning to let an agent learn simultaneously how to play a game and when to end a game.

The task is to find a single target in a grid of locations. At each time step, the agent needs to make a series of decisions:

1. It believes the target is at the currently inspected location. End the trial and see whether the result is correct.
2. It believes the target is not at the currently inspected location. It then needs to pick another location to check at the next timestep.

If the agent is choose decision #2, the environment will give some hints on where the target is, with some stochastic noise. The noise level depends on the distance between the true target location and currently inspected location. The shorter the distance, the lower the noise. The goal is to let the agent perform the task as fast and accurately as possible, so the agent needs to learn when to stop the trial, and how to select the next inspected location given the hints. The agent also has a internal memory so it won't select previously inspected locations. I would like to compare the agent's speed-accuracy trade off to human's.

In a previous simplified version of the task, the environment ends the trial once the agent hit the target location, so the agent only needs to learn how to choose the next location to inspect. I used a simple Q-network and it works well. I also found that the network should be a fully convolutional network because fully connected layers are not spatially shift-invariant.

Now how can I modify the existing convolutional network to satisfy the new task requirement? Or should I use a new network architecture?

Answer 1

I assume your agent also has to choose which locations to visit next. If so, then there are two rough designs that crossed my mind.

You can use separate agents, one for choosing to inspect or not, and one to choose which adjacent cell to visit. Sum all of the log likelihoods of both agents' actions for the loss. One particular benefit of this design is that if you can prepare the data, you can separately train the agents for awhile, and maybe train them jointly afterwards. See if the separate pre-training help improve the performance.

Other choice is to pad the "inspect current location" action alongside the "location selecting" actions, #1 inspect current location, #$2$ until #($N+1$) visit next location, given $N$ is the number of possible locations to visit.

The challenge of the two designs is how to represent this inspect action or the raw features of this action(?). If you have a domain knowledge from the game for this, maybe it can help you design it. Else you can just try with dummy features for the inspect action (maybe all zeroes) with the same length of the locations' features.