I am attempting a project involving training an agent to play a game using deep reinforcement learning. This project has a few features that complicate the design of the neural network:

  • The action space per state is very large (can be over 1000 per state)
  • The set of actions available in each state very wildly between states, both in size and the actions available.
  • The total action-space (the union of each state's action-space) is way too large to enumerate.
  • The action space is discrete, not continuous.
  • The game is adversarial, multi-agent.

Most RL neural networks I've seen involve the input of a state, and an output of a constance action size, where each element of the output is either an action's q-score or probability. But since my game's action space non-constant per state, I believe this design will not work for this game.

I have seen an alpha-go style network, which outputs a probability for all actios ever possible, and then zeros out actions not possible in the given state and re-normalized the probabilities. However, since the total action-space in my game is way to large to enumerate, I don't believe this solution will work either.

I have seen several network designs for large, discrete action spaces, including:

  • design a network to input a state-action pair and output a single score value, and train it via a value-based algorithm (such as q-learning). To select an action given a state, pass in every possible state-action pair to get each action's score, then select the action with the highest score.
  • (Wolpertinger architecture) have a network output a continous embedding, which is then mapped to a discrete action, and train it via deterministic policy gradient.
  • divide actions into a sequence of simpler actions, and train an RNN to output a sequence of these simpler actions. Train this network via a value-based algorithm (such as q-learning).

However, all of these solutions are designed for either value-based or deterministic policy gradient algorithms; none of them output probabilities over the action space. This seems to be an issue since at least a very large portion of the multi-agent deep-RL algorithms I've seen involve a network that outputs a probability over the action-space. Therfore, I don't want to limit myself to value-based and deterministic-policy algorithms.

How can I design a neural network that outputs a probability over the action space for my game? If not, what would be some good solutions to this problem?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.