Different action spaces for different states of the environment

Question

I'm new to this AI/Machine Learning and was playing around with OpenAI Gym a bit. When looking through the environments I came across "Blackjack-v0" which is a basic implementation of the game where the state is the hand count of the player and the dealer and if the player has an useable ace. The actions are only hit or stand and the possible rewards 1 if the player wins, -1 if the player loses and 0 when draw.

So that got me thinking what a more realistic environment/model for this game would look like, taking into account the current balance and other factors and has multiple actions like betting 1-10€ and hit or stand.

This brings me to my actual question:

• As far as I understand neural networks (and I do not very well yet, I guess) the input will be the state and the output the possible actions and how good the network thinks they are/will be. But now there are two different action spaces which apply to different states of the game (betting or playing), so some of the actions are useless. How would be the right way to approach this scenario?

I'm guessing one answer would be to give some kind of negative reward if the network guesses an useless action but in this case I think the reward should be the actual stake (negative reward) and the actual win if any. Therefor this would cause some bias in how the game proceeds as it should start with some amount of balance and end if the balance is 0 or after a specified amount of rounds.

Limiting timesteps wouldn't be an option either I guess because it should be limited to rounds so it won't end after a betting step e.g.

Therefore, for a useless step the reward would be 0 and the state would stay the same but for the network it doesn't matter how many useless steps it takes because it'll make no difference to the actual outcome.

Corollary question:

• Should be split up into two neural networks? One for betting and one for playing?

Answer 1

There are so many reinforcement learning algorithms. I recommend you this articles:

• Introduction to Various Reinforcement Learning Algorithms. Part 1
• Introduction to Various Reinforcement Learning Algorithms. Part 2

There are several approaches. A NN can get state and action as an input and expected discounted reward as an output. But you can just not pick an action if this action isn't available and there isn't necessity to give it negative or zero reward just don't learn your net to change it behavior for these actions because they are neither good nor bad.

Answer 2

As far as I understand from the question, the OP is trying to modeling a blackjack strategy with neural networks and asks for a certain layout to implement a multi-modal decision (betting vs. playing). The problem is, that neural networks needs something against which they can be modeled, neural networks alone are nothing and everything. Before a neural network can be implemented in Torch the underlying strategy must be programmed in normal sourcecode. The easiest way in doing so is to observe the actions of a human player. A game log is used, which is parsed by the blackjack-strategy model (programmed in sourcecode not as neural network) and only for the second step (dealing with uncertainty) a DQN neural network can be used.

Let us go into the details. In neural network based gameplaying there are two options available: model-free learning and model-based learning. The second one is easier for beginner. The model (a blackjack Game AI) is programmed with traditional programming techniques like Behavior trees, and the neural network acts as a helper modul for deciding one parameter in the hand-crafted model. For example, if in the sourcecode a variable is unknown and must adjusted to optimize a certain goal, then the neural network can help to find the parameter.

In contrast, so called model free reinforcement learning is very difficult to implement. That would be equal to start with no prior knowledge and let the network learn everything. That is often done with NEAT neuroevolutuion but i would guess, that Blackjack is a too complicated domain to learn the complete model from scratch.