How to deal with different actions for different states of the environment?

Question

I'm new to this AI/Machine Learning and was playing around with OpenAI Gym a bit. When looking through the environments, I came across the Blackjack-v0 environment, which is a basic implementation of the game where the state is the hand count of the player and the dealer and if the player has a useable ace. The actions are only hit or stand and the possible rewards 1 if the player wins, -1 if the player loses, and 0 when it is a draw.

So, that got me thinking what a more realistic environment/model for this game would look like, taking into account the current balance and other factors and has multiple actions like betting 1-10€ and hit or stand.

This brings me to my actual question:

• As far as I understand, in neural networks (and I do not very well yet, I guess) the input will be the state and the output the possible actions and how good the network thinks they are/will be. But now there are two different action spaces, which apply to different states of the game (betting or playing), so some of the actions are useless. How would be the right way to approach this scenario?

I'm guessing one answer would be to give some kind of negative reward if the network guesses a useless action, but, in this case, I think the reward should be the actual stake (negative reward) and the actual win if any. Therefore, this would cause some bias in how the game proceeds as it should start with some amount of balance and end if the balance is 0 or after a specified amount of rounds.

Limiting timesteps wouldn't be an option either, I guess, because it should be limited to rounds, so, for example, it won't end after a betting step.

Therefore, for a useless step, the reward would be 0 and the state would stay the same, but, for the neural network, it doesn't matter how many useless steps it takes because it'll make no difference to the actual outcome.

Corollary question:

• Should be split up into two neural networks? One for betting and one for playing?

Answer 1

There are so many reinforcement learning algorithms. I recommend you this articles:

• Introduction to Various Reinforcement Learning Algorithms. Part 1
• Introduction to Various Reinforcement Learning Algorithms. Part 2

There are several approaches. A NN can get state and action as an input and expected discounted reward as an output. But you can just not pick an action if this action isn't available and there isn't necessity to give it negative or zero reward just don't learn your net to change it behavior for these actions because they are neither good nor bad.