5
$\begingroup$

I've developed a neural network that can play a card game. I now want to use it to create decks for the game. My first thought would be to run a lot of games with random decks and use some approximation (maybe just a linear approximation with a feature for each card in your hand) to learn the value function for each state.

However, this will probably take a while, so in the mean time is there any way I could get this information directly from the neural network?

$\endgroup$
  • 3
    $\begingroup$ What kind of neural network do you have / how did you train it? What outputs does it have (did you train a network to output values with an algorithm like DQN, or did you train it to output a policy using policy gradient methods, or something else?) $\endgroup$ – Dennis Soemers Aug 20 '18 at 18:17
  • 2
    $\begingroup$ This may or may not be relevant, but I'm personally interested in what type of card game it is. Deck building implies some form of post-Garfield CCG. Algorithmic deck building is an exciting prospect! Can I ask, how many cards in the card "library" which a deck can be built from? $\endgroup$ – DukeZhou Aug 20 '18 at 18:33
  • 1
    $\begingroup$ @DennisSoemers There is 1 output for every action that could ever be played (almost all of this is playing card X from your hand), the highest output that is a valid move is selected to play. I used PPO to train the network. $\endgroup$ – Ryxuma Aug 21 '18 at 10:09
  • 1
    $\begingroup$ @DukeZhou There are about 500 cards in the library right now. $\endgroup$ – Ryxuma Aug 21 '18 at 10:10
4
$\begingroup$

I don't think your network, trained using PPO to play a card game, already contains sufficient information to also use for drafting. I'm not saying this with 100% certainty, maybe there's something I'm overlooking, but I can't think of anything right now.

A small adaptation to the network might be sufficient (though it would also involve re-training again). Recently, OpenAI has been writing about their attempts to train agents to play the game DOTA 2. Now, this isn't a card game, it doesn't require deckbuilding, but there is an aspect to the game that is somewhat similar to deckbuilding: drafting. In DOTA 2, there are two teams of 5 players each. Before a game start, each team selects 5 heroes (one per player) to play in that game. This is very similar to deckbuilding, except that it's likely a much smaller problem; there's only a "deck" (team composition) of 5 "cards" (heroes).

Anyway, they also trained agents to play the game (controlling one hero per agent) using PPO. In a blog post, they write the following about how they managed to add drafting capabilities relatively easily:

In late June we added a win probability output to our neural network to introspect what OpenAI Five is predicting. When later considering drafting, we realized we could use this to evaluate the win probability of any draft: just look at the prediction on the first frame of a game with that lineup. In one week of implementation, we crafted a fake frame for each of the 11 million possible team matchups and wrote a tree search to find OpenAI Five’s optimal draft.

So, if you want to try a similar technique, you'd have to adapt your network such that it also learns to generate a prediction of the win probability as output. I imagine that it'd be much less effective for deckbuilding, because win probabilities may all be very close to 50% in card games where luck (when drawing cards for example) can be a significant factor, but it might be worth a try.


Alternatively, instead of generating lots of random decks and playing with them all, you could view the problem of deckbuilding as an additional separate "game" or Markov Decision Process; adding a specific card to the deck can be an action, and this MDP terminates once you have a complete deck. Then you can try to do that better than random using search algorithms (like Monte-Carlo Tree Search) or, again, a Reinforcement Learning approach like PPO. Again, I imagine it will be a very difficult problem though, likely requiring lots of time before it will be capable of doing better than random.


I also know of some research related to deckbuilding in the collectible card game Hearthstone, which may be relevant for you. Unfortunately I did not yet get to read through any of this in detail, so I don't know for sure if you'll find a solution here, but it may be worth a try:

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.