1
$\begingroup$

For my school project, I have to develop an agent to play my game.

The Board Game

The base I have is a 'GameManager' which call 2 AIs, each taking a random move to do.

To make my AI perform, I decided to make a deep RL algorithm.

Here is how I've designed my solution.

1st : the board is a 8x8 board. making 112 possible lines to draw. 2nd : on each decision, my Agent has to choose 1 line in the remaining one. 3rd : each decision the Agent take is one among 112 possible.

I read some codes on the internet, the most relevant for me was a 'CartPole' example, which is a cart we have to slide to prevent a mass to fall.

I made an architecture which is this one: a game is simulated: the board is clean, making all 112 possibilities available. Our Agent is interroged by the gameManager to make a move passing him the actual state of the game (the state shape is a 112*1 vector of Boolean values, 1 means a line can be drawn, 0 means there is already a line on this position) (the action shape is a vector of 112*1 Boolean values, All values are set to 'False' except the line we want to draw) So, our Agent return his move decision.

Each time our agent perform a move, i store the initial state, the action we take, the reward we get performing the action, the state we reach and a boolean to know if the game is done or not.

The rewards I choose are: +1 if our action make us close a box, -1 if our action make other close a box, +10 if our action make us win the game, -10 if our action make us loose the game

The point is it's my 1st Deep learning project and I'm not sure about the mecanism i'm doing. when i launch a simulation, the Neural Network is running, but the move he does seems not to be better and better.

I give you the code I've wrote:

Here is the gameManager code:

while True:
hasMadeABox = False
gameIsEnd = False
rewardFCB = 0
doneFCB = False


if GRAPHIC_MODE:
    for event in pygame.event.get():
        if event.type == pygame.QUIT:
            pygame.quit()
            sys.exit()
    disp_board()

if playerTurns=="1":
    stateFCB = possibilitiesToBoolList(possible_moves[0])

boolArrayPossibleMoves = possibilitiesToBoolList(possible_moves[0])

theAction = players[playerTurns].play(boxes, possible_moves[0], boolArrayPossibleMoves, False)
#print(possible_moves[0])
#print(theAction)
if playerTurns =="1":
    actionFCB = theAction

if playerTurns=="1":
    is_box = move(True, theAction)

elif playerTurns=="2":
    is_box = move(False, theAction)

if is_box:
    if playerTurns =="1":
        #rewardFCB = 1
        rewardFCB = 1
        pass
    else:
        rewardFCB = -1
    hasMadeABox = True

if check_complete():
    gameIsEnd = True
    rewardFCB += 10 if score[0]>score[1] else -10 #does loosing is a reward null or negativ ?
    queueOfLastGame.pop(0)

    #Scotch pour affichage winrate
    isWin = 1 if score[0]>score[1] else -1
    queueOfLastGame.append(isWin)
    if queueOfLastGame.count(-1)+queueOfLastGame.count(1) > 0:
        print(queueOfLastGame.count(1)/(queueOfLastGame.count(-1)+queueOfLastGame.count(1)) * 100 , " % Winrate")

    doneFCB = True


if playerTurns=="1" and hasMadeABox:
    #si c'est notre IA vient de faire un carré
    #on connait directement l'état qui succede
    nextStateFCB = possibilitiesToBoolList(possible_moves[0])

if playerTurns=="2":
    nextStateFCB = possibilitiesToBoolList(possible_moves[0])


if nextStateFCB is not None:
    bufferSARS.append([stateFCB, actionFCB, rewardFCB, nextStateFCB, doneFCB])
    #ai_player_1.remember(stateFCB, actionFCB, rewardFCB, nextStateFCB, doneFCB)
    rewardFCB = 0
    nextStateFCB = None

if gameIsEnd:
    flushBufferSARS()
    reset()
    continue

#switch user to play if game is not end
if not hasMadeABox:
    playerTurns="1" if playerTurns == "2" else "2"

And here's my code about the Agent:

class Agent:
def __init__(self, name, possibleActions, stateSize, actionSize, isHuman=False, alpha=0.001, alphaDecay=0.01, batchSize=2048, learningRate=0.1, epsilon= 0.9, gamma = 0.996, hasToTrain=True):

    self._memory = deque(maxlen=100000)
    self._actualEpisode=1
    self._episodes=7000
    self._name=name
    self._possibleAction=possibleActions
    self._isHuman=isHuman
    self._epsilon=epsilon
    self._epsilonDecay = 0.99
    self._epsilonMin = 0.05
    self._gamma=gamma
    self._stateSize=stateSize
    self._actionSize=actionSize
    self._alpha=alpha
    self._alphaDecay=alphaDecay
    self._hasToTrain=hasToTrain
    self._batchSize=batchSize

    self._totalIllegalMove = 0
    self._totalLegalMove = 0

    self._path = "./modelWeightSave/"

    self._model = self._buildModel()


def save_model(self):
    self._model.save(self._path)

def getName(self):
    return self._name

def _buildModel(self):
    model = Sequential()

    model.add(Dense(128, input_dim=self._stateSize, activation='relu'))
    model.add(Dense(256, kernel_initializer='normal', activation='relu'))
    model.add(Dense(256, kernel_initializer='normal', activation='relu'))
    model.add(Dense(256, kernel_initializer='normal', activation='relu'))
    model.add(Dense(128, kernel_initializer='normal', activation='relu'))
    model.add(Dense(self._actionSize, kernel_initializer='normal', activation='relu'))

    model.compile(loss='categorical_crossentropy', optimizer=Adam(lr=self._alpha), metrics=['accuracy'])
    #if os.path.isfile(self._path):
        #model.load_weights(self._path)
    return model

def act(self, UNUSED_state, stateAsBool):
    playableIndexes = []
    for i in range(len(stateAsBool[0])):
        if stateAsBool[0][i] == 1:
            playableIndexes.append(i)
    indexForRand = playableIndexes[random.randint(0, len(playableIndexes) - 1)]

    if np.random.random() <= self._epsilon:
        action= [0]*self._actionSize
        action[indexForRand]=1

    else:
        arrayState = np.array(stateAsBool)

        action = self._model.predict(arrayState)
        #Set index of max esperence to 1, we play this line.
        tmp=[0]*self._actionSize
        tmp[np.argmax(action)] = 1
        action = tmp

        isLegalMove = True
        if sum(action) != 1:
            isLegalMove = False
        for i in range(len(action)):
            if action[i] == 1:
                if stateAsBool[0][i] == 0:
                    isLegalMove = False
                    break

        if isLegalMove:
            pass
            #print("Legal move")
        else:
            #print("Illegal move")
            #AI try to play on an already draw line, we choose a random line in remainings
            self._totalIllegalMove+=1
            action = [0] * self._actionSize
            action[indexForRand] = 1

    #print("My AI took action : ",action)
    return action

def remember(self, state, action, reward, nextState, done):
    self._memory.append((state.copy(), action, reward, nextState, done))

    self._actualEpisode+=1
    if self._actualEpisode > self._episodes:
        self._actualEpisode = 0
        self.replay(self._batchSize)

def replay(self, batchSize):
    x_batch, y_batch = [], []
    minibatch = random.sample(self._memory, min(len(self._memory), self._batchSize))
    for state, action, reward, next_state, done in minibatch:
        actionIndex = np.argmax(action)
        y_target = self._model.predict(state)
        y_target[0][actionIndex] = reward if done else reward + self._gamma * np.max(self._model.predict(next_state)[0])
        x_batch.append(state[0])
        y_batch.append(y_target[0])
    self._model.fit(np.array(x_batch), np.array(y_batch),epochs=10, batch_size=len(x_batch), verbose=1)

    if self._epsilon > self._epsilonMin:
        self._epsilon *= self._epsilonDecay

    self.save_model()

def play(self, board, state, statesAsBool, player):
    actionTaken= self.act(state, statesAsBool)
    return actionTaken

def callBackOnPreviousMove(self, state, action, reward, nextState, done):
    self.remember(state, action, reward, nextState, done)

Example of output i have during fit method:

Epoch 1/10 

1/1 [==============================] - 0s 0s/step - loss: 109.9612 - accuracy: 0.8867
 
Epoch 2/10 

1/1 [==============================] - 0s 998us/step - loss: 109.9467 - accuracy: 0.8867 

Epoch 3/10 

1/1 [==============================] - 0s 0s/step - loss: 109.9456 - accuracy: 0.8867 

Epoch 4/10 

1/1 [==============================] - 0s 0s/step - loss: 109.9332 - accuracy: 0.8867 

Epoch 5/10 

1/1 [==============================] - 0s 998us/step - loss: 109.9339 - accuracy: 0.8867 

Epoch 6/10 

1/1 [==============================] - 0s 0s/step - loss: 109.9337 - accuracy: 0.8867 

Epoch 7/10 

1/1 [==============================] - 0s 997us/step - loss: 109.9305 - accuracy: 0.8867 

Epoch 8/10 

1/1 [==============================] - 0s 0s/step - loss: 109.9314 - accuracy: 0.8867 

Epoch 9/10 

1/1 [==============================] - 0s 0s/step - loss: 109.9306 - accuracy: 0.8867 

Epoch 10/10

1/1 [==============================] - 0s 0s/step - loss: 109.9301 - accuracy: 0.8867

My questions are:

  1. Is my architecture good (inputs = [0,0,1,1,0,0,1,0.....,1,0] (112x1 shape) to represent the state, and
    output = [0,0,0,0,0,0,0,0,0,1,0,0,0...0,0,0,0] (112x1 shape with only one '1') ) to represent an action ?

  2. How to nicely choose the architecture of the Neural Network model (self._model) (I have only the basics of Neural Network, so I don't really know all activation fonction, how to design the hiden layers, choose a loss...)

  3. To train my NN, is it good to call the 'fit' function with (state, action) as parameter to make it learn?

  4. Is there something really important I forget in my design to make it work?

$\endgroup$
2
  • $\begingroup$ Please, ask one question per post. If you have multiple questions, ask each of them in their separate post. $\endgroup$ – nbro Nov 4 '20 at 18:14
  • $\begingroup$ they are highly linked :/ post 4 time the same description with only a different question should be a waste of time I think here, but i'll try to make differents posts for the next time, thanks $\endgroup$ – Benjamin Darras Nov 4 '20 at 18:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.