How to use DQN to handle an imperfect but complete information game?

Question

I'm currently having troubles to win against a random bot playing the Schieber Jass game. It is a imperfect card information game. (famous in switzerland https://www.schieber.ch/)

The environement I'm using is on Github https://github.com/murthy10/pyschieber

To get a brief overview of the Schieber Jass I will describe the main characteristics of the game. The Schieber Jass consists of four players building two teams. At the beginning every player gets randomly nine cards (there are 36 cards). Now there are nine rounds and every player has to chose one card every round. Related to the rules of the game the "highest card" wins and the team gets the points. Hence the goal is to get more points then your opponent team.

There are several more rules but I think you can image how the game should roughly work.

Now I'm trying to apply a DQN approach at the game.

To my attempts:

• I let two independent reinforcement player play against two random players
• I design the input state as a vector (one hot encoded) with 36 "bits" for every player and repeated this nine times for every card you can play during a game.
• The output is a vector of 36 "bits" for every possible card.
• If the greedy output of the network suggest an invalid action I take the action with the highest probability of the allowed actions
• The reward is +1 for winning, -1 for losing, -0.1 for a invalid action and 0 for an action which doesn't lead to a terminal state

My question:

• Would it be helpful to use a LSTM and reduce the input state?
• How to handle invalid moves?
• Do you have some good ideas for improvements? (like Neural-Fictitious Self-Play or something similar)
• Or is this the whole approach absolute nonsense?

Answer 1

Would it be helpful to use a LSTM and reduce the input state?

I'd bet, no. LSTM is more complicated and harder to learn, while the input is 4 * 9 * 36 bits is still rather limited.

However, you may want to aggregate the information somehow, e.g., add additional bits informing about what cards were already played (no matter when). This information is redundant, but by providing it, you may save the network quite some learning.

At the same time, you may want to use symmetries (all colors but trumps are equivalent and the therefore the weights should be the same).

How to handle invalid moves?

That's simple: There are no invalid moves. The network provides 36 outputs of how much it wants to play a given card. You simply take the one valid card having the greatest output value. You don't try to make the network learn what moves are valid as this is neither needed nor helpful.

Do you have some good ideas for improvements? (like Neural-Fictitious Self-Play or something similar)

I can't tell. But it should matter at the moment. First make you network clearly beat the random players, then you can look for more. Or start with Self-Play, as you want probably have both for comparison.

Or is this the whole approach absolute nonsense?

I don't think so, but ... (see below)

I design the input state as a vector (one hot encoded) with 36 "bits" for every player

This doesn't sound good. Every player has 9 of 36 cards and so should be the encoding. A player doesn't know the cards of other players.

The reward is +1 for winning, -1 for losing,

In most card games I know, it matters by how much you win (unlike e.g., in Go). Even when it doesn't matter, using this information at the early learning stages is IMHO useful.

-0.1 for a invalid action and

Drop the invalid action. Just transform anything the network produces to a valid action, add no penalty (as written above).

... 0 for an action which doesn't lead to a terminal state

All action but the last one lead to a non-terminal state. You can use some Temporal Difference Learning or use the fact that the game has a small fixed number of moves and reward/punish all actions takes in a the whole game.