I am attempting a project involving training an agent to play a game using deep reinforcement learning. This project has a few features that complicate the design of the neural network:
- The action space per state is very large (can be over 1000 per state)
- The set of actions available in each state very wildly between states, both in size and the actions available.
- The total action-space (the union of each state's action-space) is way too large to enumerate.
- The action space is discrete, not continuous.
- The game is adversarial, multi-agent.
Most RL neural networks I've seen involve the input of a state, and an output of a constance action size, where each element of the output is either an action's q-score or probability. But since my game's action space non-constant per state, I believe this design will not work for this game.
I have seen an alpha-go style network, which outputs a probability for all actios ever possible, and then zeros out actions not possible in the given state and re-normalized the probabilities. However, since the total action-space in my game is way to large to enumerate, I don't believe this solution will work either.
I have seen several network designs for large, discrete action spaces, including:
- design a network to input a state-action pair and output a single score value, and train it via a value-based algorithm (such as q-learning). To select an action given a state, pass in every possible state-action pair to get each action's score, then select the action with the highest score.
- (Wolpertinger architecture) have a network output a continous embedding, which is then mapped to a discrete action, and train it via deterministic policy gradient.
- divide actions into a sequence of simpler actions, and train an RNN to output a sequence of these simpler actions. Train this network via a value-based algorithm (such as q-learning).
However, all of these solutions are designed for either value-based or deterministic policy gradient algorithms; none of them output probabilities over the action space. This seems to be an issue since at least a very large portion of the multi-agent deep-RL algorithms I've seen involve a network that outputs a probability over the action-space. Therfore, I don't want to limit myself to value-based and deterministic-policy algorithms.
How can I design a neural network that outputs a probability over the action space for my game? If not, what would be some good solutions to this problem?