4
$\begingroup$

I'm new to this AI/Machine Learning and was playing around with OpenAI Gym a bit. When looking through the environments I came across "Blackjack-v0" which is a basic implementation of the game where the state is the hand count of the player and the dealer and if the player has an useable ace. The actions are only hit or stand and the possible rewards 1 if the player wins, -1 if the player loses and 0 when draw.

So that got me thinking what a more realistic environment/model for this game would look like, taking into account the current balance and other factors and has multiple actions like betting 1-10€ and hit or stand.

This brings me to my actual question:

  • As far as I understand neural networks (and I do not very well yet, I guess) the input will be the state and the output the possible actions and how good the network thinks they are/will be. But now there are two different action spaces which apply to different states of the game (betting or playing), so some of the actions are useless. How would be the right way to approach this scenario?

I'm guessing one answer would be to give some kind of negative reward if the network guesses an useless action but in this case I think the reward should be the actual stake (negative reward) and the actual win if any. Therefor this would cause some bias in how the game proceeds as it should start with some amount of balance and end if the balance is 0 or after a specified amount of rounds.

Limiting timesteps wouldn't be an option either I guess because it should be limited to rounds so it won't end after a betting step e.g.

Therefore, for a useless step the reward would be 0 and the state would stay the same but for the network it doesn't matter how many useless steps it takes because it'll make no difference to the actual outcome.

Corollary question:

  • Should be split up into two neural networks? One for betting and one for playing?
$\endgroup$
  • $\begingroup$ Welcome to AI! I took the liberty of massaging your question slightly, and adding the "ai-basics" tag. $\endgroup$ – DukeZhou Mar 6 '18 at 17:30
  • $\begingroup$ @DukeZhou thanks a lot for the welcome and for editing my question, I'll try to do it better next time so no massaging will be needed :) $\endgroup$ – SomeDudeCalledMo Mar 6 '18 at 17:52
  • $\begingroup$ No worries. Always happy to assist. Don't be shy about asking any informal questions on AI:chat or meta. $\endgroup$ – DukeZhou Mar 6 '18 at 20:02
1
$\begingroup$

There are so many reinforcement learning algorithms. I recommend you this articles:

There are several approaches. A NN can get state and action as an input and expected discounted reward as an output. But you can just not pick an action if this action isn't available and there isn't necessity to give it negative or zero reward just don't learn your net to change it behavior for these actions because they are neither good nor bad.

$\endgroup$
  • $\begingroup$ Thanks for the links, I'm using a DQN I know there are many algos and many that are better than a DQN but I'm pretty new to this and it was easier to implement, because of more existing examples. I guess I'll have to do a lot more research until I really get all this stuff, but again thanks for the links any resource is good at the moment :) $\endgroup$ – SomeDudeCalledMo Mar 9 '18 at 19:26
0
$\begingroup$

As far as I understand from the question, the OP is trying to modeling a blackjack strategy with neural networks and asks for a certain layout to implement a multi-modal decision (betting vs. playing). The problem is, that neural networks needs something against which they can be modeled, neural networks alone are nothing and everything. Before a neural network can be implemented in Torch the underlying strategy must be programmed in normal sourcecode. The easiest way in doing so is to observe the actions of a human player. A game log is used, which is parsed by the blackjack-strategy model (programmed in sourcecode not as neural network) and only for the second step (dealing with uncertainty) a DQN neural network can be used.

Let us go into the details. In neural network based gameplaying there are two options available: model-free learning and model-based learning. The second one is easier for beginner. The model (a blackjack Game AI) is programmed with traditional programming techniques like Behavior trees, and the neural network acts as a helper modul for deciding one parameter in the hand-crafted model. For example, if in the sourcecode a variable is unknown and must adjusted to optimize a certain goal, then the neural network can help to find the parameter.

In contrast, so called model free reinforcement learning is very difficult to implement. That would be equal to start with no prior knowledge and let the network learn everything. That is often done with NEAT neuroevolutuion but i would guess, that Blackjack is a too complicated domain to learn the complete model from scratch.

$\endgroup$
  • $\begingroup$ Thanks a lot for your answer! I wanted to do model free learning as I wanted to see what kind of tactics the AI finds out on it's own without any guidings. Since I couldn't really figure out described problem I decided to go with multiple networks for multiple actions. So I wrote an environment similar to open ai gym, with 5 different actions (bet, insure, split, double, hit) at different timesteps. I use a DQN (openai baselines) as network for each of these actions. Now I'm still not quite sure if this is the right approach because they aren't “aware“ of the others and what these are doing $\endgroup$ – SomeDudeCalledMo Apr 10 '18 at 17:36
  • $\begingroup$ I had to squeeze this a bit because it was too long, but what I meant with them not being aware of eachother is they don't “know“ which steps reallly brought them into the state they are in now and the overall balance or reward over time. Maybe that wouldn't make any difference or even bias decisions but I'm not sure. I'll have to do some more research on the topic I guess but at least I learned a lot so far :) $\endgroup$ – SomeDudeCalledMo Apr 10 '18 at 17:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.