Teaching a neural network to play a card game with two phases

Question

I'm writing a virtual environment for a 4-player card game named estimation, and will use deep reinforcement learning to teach an agent to play it.

Each player gets a hand of 13 cards, and the first phase is for each player to estimate the number of tricks they will collect. The highest player gets to start first,and then after each round, the player who collects the trick starts the next. So basically the first phase is for bidding and the next phase consists of 13 rounds.

The state input I'll use will include all the cards that have been played, the goal and collected tricks, and the available cards. The output for each round will be a vector of length 54, containing all the cards and then the available card with the highest probability would be played.

At first I thought that the bidding phase should use the same input but with zeros everywhere except the available hand, and the output would exclude all the cards with no numbers like king, queen or jack. But then the ability to dash (estimate that you'll collect 0 tricks) wouldn't be available. Also I don't think it would work really well.

Should I just use two NNs for each phase, or what should I do? Also if anyone has any advice on things I need to watch out for, I'd really appreciate it if they shared them.

Answer 1

I've played this game. If I remember correctly, a successful strategy (for winning as many tricks you estimated to win) involves continually evaluating how well you're doing. If you're doing very poorly relative to your initial estimation, that's valuable information to have.

I think you should have one network that outputs--at every time step--both an estimate of future tricks won and an action of which card to play. This has the additional advantage of giving you more estimation experience with which to train. After all rounds, you'll know how far off each of your estimations were throughout the game. This should speed up learning on that part of the game.