I've developed a neural network that can play a card game. I now want to use it to create decks for the game. My first thought would be to run a lot of games with random decks and use some approximation (maybe just a linear approximation with a feature for each card in your hand) to learn the value function for each state.
However, this will probably take a while, so in the mean time is there any way I could get this information directly from the neural network?
I don't think your network, trained using PPO to play a card game, already contains sufficient information to also use for drafting. I'm not saying this with 100% certainty, maybe there's something I'm overlooking, but I can't think of anything right now.
A small adaptation to the network might be sufficient (though it would also involve re-training again). Recently, OpenAI has been writing about their attempts to train agents to play the game DOTA 2. Now, this isn't a card game, it doesn't require deckbuilding, but there is an aspect to the game that is somewhat similar to deckbuilding: drafting. In DOTA 2, there are two teams of 5 players each. Before a game start, each team selects 5 heroes (one per player) to play in that game. This is very similar to deckbuilding, except that it's likely a much smaller problem; there's only a "deck" (team composition) of 5 "cards" (heroes).
Anyway, they also trained agents to play the game (controlling one hero per agent) using PPO. In a blog post, they write the following about how they managed to add drafting capabilities relatively easily:
In late June we added a win probability output to our neural network to introspect what OpenAI Five is predicting. When later considering drafting, we realized we could use this to evaluate the win probability of any draft: just look at the prediction on the first frame of a game with that lineup. In one week of implementation, we crafted a fake frame for each of the 11 million possible team matchups and wrote a tree search to find OpenAI Five’s optimal draft.
So, if you want to try a similar technique, you'd have to adapt your network such that it also learns to generate a prediction of the win probability as output. I imagine that it'd be much less effective for deckbuilding, because win probabilities may all be very close to 50% in card games where luck (when drawing cards for example) can be a significant factor, but it might be worth a try.
Alternatively, instead of generating lots of random decks and playing with them all, you could view the problem of deckbuilding as an additional separate "game" or Markov Decision Process; adding a specific card to the deck can be an action, and this MDP terminates once you have a complete deck. Then you can try to do that better than random using search algorithms (like Monte-Carlo Tree Search) or, again, a Reinforcement Learning approach like PPO. Again, I imagine it will be a very difficult problem though, likely requiring lots of time before it will be capable of doing better than random.
I also know of some research related to deckbuilding in the collectible card game Hearthstone, which may be relevant for you. Unfortunately I did not yet get to read through any of this in detail, so I don't know for sure if you'll find a solution here, but it may be worth a try:
