# How to design my Neural Network for Game AI

For my school project, I have to develop an agent to play my game.

The base I have is a 'GameManager' which call 2 AIs, each taking a random move to do.

To make my AI perform, I decided to make a deep RL algorithm.

Here is how I've designed my solution.

1st : the board is a 8x8 board. making 112 possible lines to draw. 2nd : on each decision, my Agent has to choose 1 line in the remaining one. 3rd : each decision the Agent take is one among 112 possible.

I read some codes on the internet, the most relevant for me was a 'CartPole' example, which is a cart we have to slide to prevent a mass to fall.

I made an architecture which is this one: a game is simulated: the board is clean, making all 112 possibilities available. Our Agent is interroged by the gameManager to make a move passing him the actual state of the game (the state shape is a 112*1 vector of Boolean values, 1 means a line can be drawn, 0 means there is already a line on this position) (the action shape is a vector of 112*1 Boolean values, All values are set to 'False' except the line we want to draw) So, our Agent return his move decision.

Each time our agent perform a move, i store the initial state, the action we take, the reward we get performing the action, the state we reach and a boolean to know if the game is done or not.

The rewards I choose are: +1 if our action make us close a box, -1 if our action make other close a box, +10 if our action make us win the game, -10 if our action make us loose the game

The point is it's my 1st Deep learning project and I'm not sure about the mecanism i'm doing. when i launch a simulation, the Neural Network is running, but the move he does seems not to be better and better.

I give you the code I've wrote:

Here is the gameManager code:

while True:
gameIsEnd = False
rewardFCB = 0
doneFCB = False

if GRAPHIC_MODE:
for event in pygame.event.get():
if event.type == pygame.QUIT:
pygame.quit()
sys.exit()
disp_board()

if playerTurns=="1":
stateFCB = possibilitiesToBoolList(possible_moves[0])

boolArrayPossibleMoves = possibilitiesToBoolList(possible_moves[0])

theAction = players[playerTurns].play(boxes, possible_moves[0], boolArrayPossibleMoves, False)
#print(possible_moves[0])
#print(theAction)
if playerTurns =="1":
actionFCB = theAction

if playerTurns=="1":
is_box = move(True, theAction)

elif playerTurns=="2":
is_box = move(False, theAction)

if is_box:
if playerTurns =="1":
#rewardFCB = 1
rewardFCB = 1
pass
else:
rewardFCB = -1

if check_complete():
gameIsEnd = True
rewardFCB += 10 if score[0]>score[1] else -10 #does loosing is a reward null or negativ ?
queueOfLastGame.pop(0)

#Scotch pour affichage winrate
isWin = 1 if score[0]>score[1] else -1
queueOfLastGame.append(isWin)
if queueOfLastGame.count(-1)+queueOfLastGame.count(1) > 0:
print(queueOfLastGame.count(1)/(queueOfLastGame.count(-1)+queueOfLastGame.count(1)) * 100 , " % Winrate")

doneFCB = True

#si c'est notre IA vient de faire un carré
#on connait directement l'état qui succede
nextStateFCB = possibilitiesToBoolList(possible_moves[0])

if playerTurns=="2":
nextStateFCB = possibilitiesToBoolList(possible_moves[0])

if nextStateFCB is not None:
bufferSARS.append([stateFCB, actionFCB, rewardFCB, nextStateFCB, doneFCB])
#ai_player_1.remember(stateFCB, actionFCB, rewardFCB, nextStateFCB, doneFCB)
rewardFCB = 0
nextStateFCB = None

if gameIsEnd:
flushBufferSARS()
reset()
continue

#switch user to play if game is not end
playerTurns="1" if playerTurns == "2" else "2"

And here's my code about the Agent:

class Agent:
def __init__(self, name, possibleActions, stateSize, actionSize, isHuman=False, alpha=0.001, alphaDecay=0.01, batchSize=2048, learningRate=0.1, epsilon= 0.9, gamma = 0.996, hasToTrain=True):

self._memory = deque(maxlen=100000)
self._actualEpisode=1
self._episodes=7000
self._name=name
self._possibleAction=possibleActions
self._isHuman=isHuman
self._epsilon=epsilon
self._epsilonDecay = 0.99
self._epsilonMin = 0.05
self._gamma=gamma
self._stateSize=stateSize
self._actionSize=actionSize
self._alpha=alpha
self._hasToTrain=hasToTrain
self._batchSize=batchSize

self._totalIllegalMove = 0
self._totalLegalMove = 0

self._path = "./modelWeightSave/"

self._model = self._buildModel()

def save_model(self):
self._model.save(self._path)

def getName(self):
return self._name

def _buildModel(self):
model = Sequential()

#if os.path.isfile(self._path):
return model

def act(self, UNUSED_state, stateAsBool):
playableIndexes = []
for i in range(len(stateAsBool[0])):
if stateAsBool[0][i] == 1:
playableIndexes.append(i)
indexForRand = playableIndexes[random.randint(0, len(playableIndexes) - 1)]

if np.random.random() <= self._epsilon:
action= [0]*self._actionSize
action[indexForRand]=1

else:
arrayState = np.array(stateAsBool)

action = self._model.predict(arrayState)
#Set index of max esperence to 1, we play this line.
tmp=[0]*self._actionSize
tmp[np.argmax(action)] = 1
action = tmp

isLegalMove = True
if sum(action) != 1:
isLegalMove = False
for i in range(len(action)):
if action[i] == 1:
if stateAsBool[0][i] == 0:
isLegalMove = False
break

if isLegalMove:
pass
#print("Legal move")
else:
#print("Illegal move")
#AI try to play on an already draw line, we choose a random line in remainings
self._totalIllegalMove+=1
action = [0] * self._actionSize
action[indexForRand] = 1

#print("My AI took action : ",action)
return action

def remember(self, state, action, reward, nextState, done):
self._memory.append((state.copy(), action, reward, nextState, done))

self._actualEpisode+=1
if self._actualEpisode > self._episodes:
self._actualEpisode = 0
self.replay(self._batchSize)

def replay(self, batchSize):
x_batch, y_batch = [], []
minibatch = random.sample(self._memory, min(len(self._memory), self._batchSize))
for state, action, reward, next_state, done in minibatch:
actionIndex = np.argmax(action)
y_target = self._model.predict(state)
y_target[0][actionIndex] = reward if done else reward + self._gamma * np.max(self._model.predict(next_state)[0])
x_batch.append(state[0])
y_batch.append(y_target[0])
self._model.fit(np.array(x_batch), np.array(y_batch),epochs=10, batch_size=len(x_batch), verbose=1)

if self._epsilon > self._epsilonMin:
self._epsilon *= self._epsilonDecay

self.save_model()

def play(self, board, state, statesAsBool, player):
actionTaken= self.act(state, statesAsBool)
return actionTaken

def callBackOnPreviousMove(self, state, action, reward, nextState, done):
self.remember(state, action, reward, nextState, done)

Example of output i have during fit method:

Epoch 1/10

1/1 [==============================] - 0s 0s/step - loss: 109.9612 - accuracy: 0.8867

Epoch 2/10

1/1 [==============================] - 0s 998us/step - loss: 109.9467 - accuracy: 0.8867

Epoch 3/10

1/1 [==============================] - 0s 0s/step - loss: 109.9456 - accuracy: 0.8867

Epoch 4/10

1/1 [==============================] - 0s 0s/step - loss: 109.9332 - accuracy: 0.8867

Epoch 5/10

1/1 [==============================] - 0s 998us/step - loss: 109.9339 - accuracy: 0.8867

Epoch 6/10

1/1 [==============================] - 0s 0s/step - loss: 109.9337 - accuracy: 0.8867

Epoch 7/10

1/1 [==============================] - 0s 997us/step - loss: 109.9305 - accuracy: 0.8867

Epoch 8/10

1/1 [==============================] - 0s 0s/step - loss: 109.9314 - accuracy: 0.8867

Epoch 9/10

1/1 [==============================] - 0s 0s/step - loss: 109.9306 - accuracy: 0.8867

Epoch 10/10

1/1 [==============================] - 0s 0s/step - loss: 109.9301 - accuracy: 0.8867

My questions are:

1. Is my architecture good (inputs = [0,0,1,1,0,0,1,0.....,1,0] (112x1 shape) to represent the state, and
output = [0,0,0,0,0,0,0,0,0,1,0,0,0...0,0,0,0] (112x1 shape with only one '1') ) to represent an action ?

2. How to nicely choose the architecture of the Neural Network model (self._model) (I have only the basics of Neural Network, so I don't really know all activation fonction, how to design the hiden layers, choose a loss...)

3. To train my NN, is it good to call the 'fit' function with (state, action) as parameter to make it learn?

4. Is there something really important I forget in my design to make it work?

• Please, ask one question per post. If you have multiple questions, ask each of them in their separate post. – nbro Nov 4 '20 at 18:14
• they are highly linked :/ post 4 time the same description with only a different question should be a waste of time I think here, but i'll try to make differents posts for the next time, thanks – Benjamin Darras Nov 4 '20 at 18:18