DQN card game - how to represent the actions?

Question

I want to train a DQN card game named witches. It consists of 60 Cards (1-14 of Yellow, Blue, Green, Red Cards + 4 Wizzards). The color of the first layed card has to be respected by the other players (if they have this card in hand). The one who has the card with the highest number gets the played cards. Each collected red card gives you a -1 point.

With respect to this answer I setup the inputs / state of the NN as binary (meaning I have 180 bool values (60 card x is currently on the table, 60 card x is in the ai hand, 60 card x is already been played)

How to design the outputs / actions?

• If the ai is the first player of a round it can play any card
• If the ai is not first player it has to respect the first played card (or play a wizzard)

This means there is actually a list of available options. I then sort this list and have 60 Output bools which I set to 1 if this option is possible. Among these options the ai should then decide what the correct option is? Is this the correct procedure?

Inconsistent / Varying Action Space This is what we have to deal here with. As explained in here I think a DQN as well as Policy Gradient Methods is not the correct architecture to choose for solving such multi-agent card games. What architecture would you choose?

General procedure?

Assume I have 4 players, so do I have to get the old state before the ai is the next player and the new state is directly after this round is finished?

my_game = game(["Tim", "Bob", "Lena", "Anja"])
while True:
    #1. Play unti AI has to move:
    my_game.play_round_until_ai()

    #2. Get old state:
    state_old = agent.get_state(my_game)

    #3. Get the action the AI should perform
    action_ = agent.get_action(state_old, my_game)

    #4. perform new Action and get new state
    #reward rates how good the current action was
    #score is the actual score of this game!
    reward, done, score = my_game.finishRound(action_)

    # 5: Calculate new state
    state_new = agent.get_state(my_game)

    #6. train short memory base on the new action and state
    agent.train_short_memory(state_old, action_, reward, state_new, done)

    #7. store the new data into a long term memory
    agent.remember(state_old, action_, reward, state_new, done)

    if done == True:
        # One game is over, train on the memory and plot the result.
        sc = my_game.reset()

My code so far is available here: https://github.com/CesMak/witches_ai

Answer 1

Because your action space is not discrete, It would not be wise to evaluate possible actions. What would probably be better is evaluating next states. This means evaluating the value of the state after the action with your neural network instead of the value of a action.

Why is this better? You can use the neural net for each possible action , instead of trying to make the neural network figure out which actions are possible. the result would be taking the action of which the next state has the most value.