For my school project, I have to develop an agent to play my game.

The Board Game

The base I have is a 'GameManager' which call 2 AIs, each taking a random move to do.

To make my AI perform, I decided to make a deep RL algorithm.

Here is how I've designed my solution.

1st : the board is a 8x8 board. making 112 possible lines to draw. 2nd : on each decision, my Agent has to choose 1 line in the remaining one. 3rd : each decision the Agent take is one among 112 possible.

I read some codes on the internet, the most relevant for me was a 'CartPole' example, which is a cart we have to slide to prevent a mass to fall.

I made an architecture which is this one: a game is simulated: the board is clean, making all 112 possibilities available. Our Agent is interroged by the gameManager to make a move passing him the actual state of the game (the state shape is a 112*1 vector of Boolean values, 1 means a line can be drawn, 0 means there is already a line on this position) (the action shape is a vector of 112*1 Boolean values, All values are set to 'False' except the line we want to draw) So, our Agent return his move decision.

Each time our agent perform a move, i store the initial state, the action we take, the reward we get performing the action, the state we reach and a boolean to know if the game is done or not.

The rewards I choose are: +1 if our action make us close a box, -1 if our action make other close a box, +10 if our action make us win the game, -10 if our action make us loose the game

The point is it's my 1st Deep learning project and I'm not sure about the mecanism i'm doing. when i launch a simulation, the Neural Network is running, but the move he does seems not to be better and better.

I give you the code I've wrote:

Here is the gameManager code:

while True:
hasMadeABox = False
gameIsEnd = False
rewardFCB = 0
doneFCB = False

    for event in pygame.event.get():
        if event.type == pygame.QUIT:

if playerTurns=="1":
    stateFCB = possibilitiesToBoolList(possible_moves[0])

boolArrayPossibleMoves = possibilitiesToBoolList(possible_moves[0])

theAction = players[playerTurns].play(boxes, possible_moves[0], boolArrayPossibleMoves, False)
if playerTurns =="1":
    actionFCB = theAction

if playerTurns=="1":
    is_box = move(True, theAction)

elif playerTurns=="2":
    is_box = move(False, theAction)

if is_box:
    if playerTurns =="1":
        #rewardFCB = 1
        rewardFCB = 1
        rewardFCB = -1
    hasMadeABox = True

if check_complete():
    gameIsEnd = True
    rewardFCB += 10 if score[0]>score[1] else -10 #does loosing is a reward null or negativ ?

    #Scotch pour affichage winrate
    isWin = 1 if score[0]>score[1] else -1
    if queueOfLastGame.count(-1)+queueOfLastGame.count(1) > 0:
        print(queueOfLastGame.count(1)/(queueOfLastGame.count(-1)+queueOfLastGame.count(1)) * 100 , " % Winrate")

    doneFCB = True

if playerTurns=="1" and hasMadeABox:
    #si c'est notre IA vient de faire un carré
    #on connait directement l'état qui succede
    nextStateFCB = possibilitiesToBoolList(possible_moves[0])

if playerTurns=="2":
    nextStateFCB = possibilitiesToBoolList(possible_moves[0])

if nextStateFCB is not None:
    bufferSARS.append([stateFCB, actionFCB, rewardFCB, nextStateFCB, doneFCB])
    #ai_player_1.remember(stateFCB, actionFCB, rewardFCB, nextStateFCB, doneFCB)
    rewardFCB = 0
    nextStateFCB = None

if gameIsEnd:

#switch user to play if game is not end
if not hasMadeABox:
    playerTurns="1" if playerTurns == "2" else "2"

And here's my code about the Agent:

class Agent:
def __init__(self, name, possibleActions, stateSize, actionSize, isHuman=False, alpha=0.001, alphaDecay=0.01, batchSize=2048, learningRate=0.1, epsilon= 0.9, gamma = 0.996, hasToTrain=True):

    self._memory = deque(maxlen=100000)
    self._epsilonDecay = 0.99
    self._epsilonMin = 0.05

    self._totalIllegalMove = 0
    self._totalLegalMove = 0

    self._path = "./modelWeightSave/"

    self._model = self._buildModel()

def save_model(self):

def getName(self):
    return self._name

def _buildModel(self):
    model = Sequential()

    model.add(Dense(128, input_dim=self._stateSize, activation='relu'))
    model.add(Dense(256, kernel_initializer='normal', activation='relu'))
    model.add(Dense(256, kernel_initializer='normal', activation='relu'))
    model.add(Dense(256, kernel_initializer='normal', activation='relu'))
    model.add(Dense(128, kernel_initializer='normal', activation='relu'))
    model.add(Dense(self._actionSize, kernel_initializer='normal', activation='relu'))

    model.compile(loss='categorical_crossentropy', optimizer=Adam(lr=self._alpha), metrics=['accuracy'])
    #if os.path.isfile(self._path):
    return model

def act(self, UNUSED_state, stateAsBool):
    playableIndexes = []
    for i in range(len(stateAsBool[0])):
        if stateAsBool[0][i] == 1:
    indexForRand = playableIndexes[random.randint(0, len(playableIndexes) - 1)]

    if np.random.random() <= self._epsilon:
        action= [0]*self._actionSize

        arrayState = np.array(stateAsBool)

        action = self._model.predict(arrayState)
        #Set index of max esperence to 1, we play this line.
        tmp[np.argmax(action)] = 1
        action = tmp

        isLegalMove = True
        if sum(action) != 1:
            isLegalMove = False
        for i in range(len(action)):
            if action[i] == 1:
                if stateAsBool[0][i] == 0:
                    isLegalMove = False

        if isLegalMove:
            #print("Legal move")
            #print("Illegal move")
            #AI try to play on an already draw line, we choose a random line in remainings
            action = [0] * self._actionSize
            action[indexForRand] = 1

    #print("My AI took action : ",action)
    return action

def remember(self, state, action, reward, nextState, done):
    self._memory.append((state.copy(), action, reward, nextState, done))

    if self._actualEpisode > self._episodes:
        self._actualEpisode = 0

def replay(self, batchSize):
    x_batch, y_batch = [], []
    minibatch = random.sample(self._memory, min(len(self._memory), self._batchSize))
    for state, action, reward, next_state, done in minibatch:
        actionIndex = np.argmax(action)
        y_target = self._model.predict(state)
        y_target[0][actionIndex] = reward if done else reward + self._gamma * np.max(self._model.predict(next_state)[0])
    self._model.fit(np.array(x_batch), np.array(y_batch),epochs=10, batch_size=len(x_batch), verbose=1)

    if self._epsilon > self._epsilonMin:
        self._epsilon *= self._epsilonDecay


def play(self, board, state, statesAsBool, player):
    actionTaken= self.act(state, statesAsBool)
    return actionTaken

def callBackOnPreviousMove(self, state, action, reward, nextState, done):
    self.remember(state, action, reward, nextState, done)

Example of output i have during fit method:

Epoch 1/10 

1/1 [==============================] - 0s 0s/step - loss: 109.9612 - accuracy: 0.8867
Epoch 2/10 

1/1 [==============================] - 0s 998us/step - loss: 109.9467 - accuracy: 0.8867 

Epoch 3/10 

1/1 [==============================] - 0s 0s/step - loss: 109.9456 - accuracy: 0.8867 

Epoch 4/10 

1/1 [==============================] - 0s 0s/step - loss: 109.9332 - accuracy: 0.8867 

Epoch 5/10 

1/1 [==============================] - 0s 998us/step - loss: 109.9339 - accuracy: 0.8867 

Epoch 6/10 

1/1 [==============================] - 0s 0s/step - loss: 109.9337 - accuracy: 0.8867 

Epoch 7/10 

1/1 [==============================] - 0s 997us/step - loss: 109.9305 - accuracy: 0.8867 

Epoch 8/10 

1/1 [==============================] - 0s 0s/step - loss: 109.9314 - accuracy: 0.8867 

Epoch 9/10 

1/1 [==============================] - 0s 0s/step - loss: 109.9306 - accuracy: 0.8867 

Epoch 10/10

1/1 [==============================] - 0s 0s/step - loss: 109.9301 - accuracy: 0.8867

My questions are:

  1. Is my architecture good (inputs = [0,0,1,1,0,0,1,0.....,1,0] (112x1 shape) to represent the state, and
    output = [0,0,0,0,0,0,0,0,0,1,0,0,0...0,0,0,0] (112x1 shape with only one '1') ) to represent an action ?

  2. How to nicely choose the architecture of the Neural Network model (self._model) (I have only the basics of Neural Network, so I don't really know all activation fonction, how to design the hiden layers, choose a loss...)

  3. To train my NN, is it good to call the 'fit' function with (state, action) as parameter to make it learn?

  4. Is there something really important I forget in my design to make it work?

  • $\begingroup$ Please, ask one question per post. If you have multiple questions, ask each of them in their separate post. $\endgroup$ – nbro Nov 4 '20 at 18:14
  • $\begingroup$ they are highly linked :/ post 4 time the same description with only a different question should be a waste of time I think here, but i'll try to make differents posts for the next time, thanks $\endgroup$ – Benjamin Darras Nov 4 '20 at 18:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.