I want to train a DQN card game named witches. It consists of 60 Cards (1-14 of Yellow, Blue, Green, Red Cards + 4 Wizzards). The color of the first layed card has to be respected by the other players (if they have this card in hand). The one who has the card with the highest number gets the played cards. Each collected red card gives you a -1 point.

With respect to this answer I setup the inputs / state of the NN as binary (meaning I have 180 bool values (60 card x is currently on the table, 60 card x is in the ai hand, 60 card x is already been played)

How to design the outputs / actions?

  • If the ai is the first player of a round it can play any card
  • If the ai is not first player it has to respect the first played card (or play a wizzard)

This means there is actually a list of available options. I then sort this list and have 60 Output bools which I set to 1 if this option is possible. Among these options the ai should then decide what the correct option is? Is this the correct procedure?

Inconsistent / Varying Action Space This is what we have to deal here with. As explained in here I think a DQN as well as Policy Gradient Methods is not the correct architecture to choose for solving such multi-agent card games. What architecture would you choose?

General procedure?

Assume I have 4 players, so do I have to get the old state before the ai is the next player and the new state is directly after this round is finished?

my_game = game(["Tim", "Bob", "Lena", "Anja"])
while True:
    #1. Play unti AI has to move:

    #2. Get old state:
    state_old = agent.get_state(my_game)

    #3. Get the action the AI should perform
    action_ = agent.get_action(state_old, my_game)

    #4. perform new Action and get new state
    #reward rates how good the current action was
    #score is the actual score of this game!
    reward, done, score = my_game.finishRound(action_)

    # 5: Calculate new state
    state_new = agent.get_state(my_game)

    #6. train short memory base on the new action and state
    agent.train_short_memory(state_old, action_, reward, state_new, done)

    #7. store the new data into a long term memory
    agent.remember(state_old, action_, reward, state_new, done)

    if done == True:
        # One game is over, train on the memory and plot the result.
        sc = my_game.reset()

My code so far is available here: https://github.com/CesMak/witches_ai


1 Answer 1


Because your action space is not discrete, It would not be wise to evaluate possible actions. What would probably be better is evaluating next states. This means evaluating the value of the state after the action with your neural network instead of the value of a action.

Why is this better? You can use the neural net for each possible action , instead of trying to make the neural network figure out which actions are possible. the result would be taking the action of which the next state has the most value.

  • $\begingroup$ Could you point me to some examples? $\endgroup$
    – sqp_125
    Commented Dec 9, 2019 at 19:36
  • 1
    $\begingroup$ sure, I found this answer on this website : ai.stackexchange.com/questions/9491/… $\endgroup$ Commented Dec 10, 2019 at 13:22
  • $\begingroup$ I just read this article but I cannot find what you are suggesting. What I could directly re-implement is choice (1): Such that the network learns possible actions but this is as said not the best option. Do you know some code example / what would be the name of this algorithm you suggested (I think it is not DQN anymore). Sry I am completely new to in this topic. $\endgroup$
    – sqp_125
    Commented Dec 10, 2019 at 15:37
  • $\begingroup$ Try looking up on state value functions vs state action value functions. I remember learning this from the book 'Reinforcement learning : an introduction'. from A quick internet search maybe this will help you understand: cs.dartmouth.edu/~lorenzo/teaching/cs134/Archive/Spring2009/… $\endgroup$ Commented Dec 11, 2019 at 9:03

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .