For my school project, I have to develop an agent to play my game.
The base I have is a 'GameManager' which call 2 AIs, each taking a random move to do.
To make my AI perform, I decided to make a deep RL algorithm.
Here is how I've designed my solution.
1st : the board is a 8x8 board. making 112 possible lines to draw. 2nd : on each decision, my Agent has to choose 1 line in the remaining one. 3rd : each decision the Agent take is one among 112 possible.
I read some codes on the internet, the most relevant for me was a 'CartPole' example, which is a cart we have to slide to prevent a mass to fall.
I made an architecture which is this one: a game is simulated: the board is clean, making all 112 possibilities available. Our Agent is interroged by the gameManager to make a move passing him the actual state of the game (the state shape is a 112*1 vector of Boolean values, 1 means a line can be drawn, 0 means there is already a line on this position) (the action shape is a vector of 112*1 Boolean values, All values are set to 'False' except the line we want to draw) So, our Agent return his move decision.
Each time our agent perform a move, i store the initial state, the action we take, the reward we get performing the action, the state we reach and a boolean to know if the game is done or not.
The rewards I choose are: +1 if our action make us close a box, -1 if our action make other close a box, +10 if our action make us win the game, -10 if our action make us loose the game
The point is it's my 1st Deep learning project and I'm not sure about the mecanism i'm doing. when i launch a simulation, the Neural Network is running, but the move he does seems not to be better and better.
I give you the code I've wrote:
Here is the gameManager code:
while True:
hasMadeABox = False
gameIsEnd = False
rewardFCB = 0
doneFCB = False
if GRAPHIC_MODE:
for event in pygame.event.get():
if event.type == pygame.QUIT:
pygame.quit()
sys.exit()
disp_board()
if playerTurns=="1":
stateFCB = possibilitiesToBoolList(possible_moves[0])
boolArrayPossibleMoves = possibilitiesToBoolList(possible_moves[0])
theAction = players[playerTurns].play(boxes, possible_moves[0], boolArrayPossibleMoves, False)
#print(possible_moves[0])
#print(theAction)
if playerTurns =="1":
actionFCB = theAction
if playerTurns=="1":
is_box = move(True, theAction)
elif playerTurns=="2":
is_box = move(False, theAction)
if is_box:
if playerTurns =="1":
#rewardFCB = 1
rewardFCB = 1
pass
else:
rewardFCB = -1
hasMadeABox = True
if check_complete():
gameIsEnd = True
rewardFCB += 10 if score[0]>score[1] else -10 #does loosing is a reward null or negativ ?
queueOfLastGame.pop(0)
#Scotch pour affichage winrate
isWin = 1 if score[0]>score[1] else -1
queueOfLastGame.append(isWin)
if queueOfLastGame.count(-1)+queueOfLastGame.count(1) > 0:
print(queueOfLastGame.count(1)/(queueOfLastGame.count(-1)+queueOfLastGame.count(1)) * 100 , " % Winrate")
doneFCB = True
if playerTurns=="1" and hasMadeABox:
#si c'est notre IA vient de faire un carré
#on connait directement l'état qui succede
nextStateFCB = possibilitiesToBoolList(possible_moves[0])
if playerTurns=="2":
nextStateFCB = possibilitiesToBoolList(possible_moves[0])
if nextStateFCB is not None:
bufferSARS.append([stateFCB, actionFCB, rewardFCB, nextStateFCB, doneFCB])
#ai_player_1.remember(stateFCB, actionFCB, rewardFCB, nextStateFCB, doneFCB)
rewardFCB = 0
nextStateFCB = None
if gameIsEnd:
flushBufferSARS()
reset()
continue
#switch user to play if game is not end
if not hasMadeABox:
playerTurns="1" if playerTurns == "2" else "2"
And here's my code about the Agent:
class Agent:
def __init__(self, name, possibleActions, stateSize, actionSize, isHuman=False, alpha=0.001, alphaDecay=0.01, batchSize=2048, learningRate=0.1, epsilon= 0.9, gamma = 0.996, hasToTrain=True):
self._memory = deque(maxlen=100000)
self._actualEpisode=1
self._episodes=7000
self._name=name
self._possibleAction=possibleActions
self._isHuman=isHuman
self._epsilon=epsilon
self._epsilonDecay = 0.99
self._epsilonMin = 0.05
self._gamma=gamma
self._stateSize=stateSize
self._actionSize=actionSize
self._alpha=alpha
self._alphaDecay=alphaDecay
self._hasToTrain=hasToTrain
self._batchSize=batchSize
self._totalIllegalMove = 0
self._totalLegalMove = 0
self._path = "./modelWeightSave/"
self._model = self._buildModel()
def save_model(self):
self._model.save(self._path)
def getName(self):
return self._name
def _buildModel(self):
model = Sequential()
model.add(Dense(128, input_dim=self._stateSize, activation='relu'))
model.add(Dense(256, kernel_initializer='normal', activation='relu'))
model.add(Dense(256, kernel_initializer='normal', activation='relu'))
model.add(Dense(256, kernel_initializer='normal', activation='relu'))
model.add(Dense(128, kernel_initializer='normal', activation='relu'))
model.add(Dense(self._actionSize, kernel_initializer='normal', activation='relu'))
model.compile(loss='categorical_crossentropy', optimizer=Adam(lr=self._alpha), metrics=['accuracy'])
#if os.path.isfile(self._path):
#model.load_weights(self._path)
return model
def act(self, UNUSED_state, stateAsBool):
playableIndexes = []
for i in range(len(stateAsBool[0])):
if stateAsBool[0][i] == 1:
playableIndexes.append(i)
indexForRand = playableIndexes[random.randint(0, len(playableIndexes) - 1)]
if np.random.random() <= self._epsilon:
action= [0]*self._actionSize
action[indexForRand]=1
else:
arrayState = np.array(stateAsBool)
action = self._model.predict(arrayState)
#Set index of max esperence to 1, we play this line.
tmp=[0]*self._actionSize
tmp[np.argmax(action)] = 1
action = tmp
isLegalMove = True
if sum(action) != 1:
isLegalMove = False
for i in range(len(action)):
if action[i] == 1:
if stateAsBool[0][i] == 0:
isLegalMove = False
break
if isLegalMove:
pass
#print("Legal move")
else:
#print("Illegal move")
#AI try to play on an already draw line, we choose a random line in remainings
self._totalIllegalMove+=1
action = [0] * self._actionSize
action[indexForRand] = 1
#print("My AI took action : ",action)
return action
def remember(self, state, action, reward, nextState, done):
self._memory.append((state.copy(), action, reward, nextState, done))
self._actualEpisode+=1
if self._actualEpisode > self._episodes:
self._actualEpisode = 0
self.replay(self._batchSize)
def replay(self, batchSize):
x_batch, y_batch = [], []
minibatch = random.sample(self._memory, min(len(self._memory), self._batchSize))
for state, action, reward, next_state, done in minibatch:
actionIndex = np.argmax(action)
y_target = self._model.predict(state)
y_target[0][actionIndex] = reward if done else reward + self._gamma * np.max(self._model.predict(next_state)[0])
x_batch.append(state[0])
y_batch.append(y_target[0])
self._model.fit(np.array(x_batch), np.array(y_batch),epochs=10, batch_size=len(x_batch), verbose=1)
if self._epsilon > self._epsilonMin:
self._epsilon *= self._epsilonDecay
self.save_model()
def play(self, board, state, statesAsBool, player):
actionTaken= self.act(state, statesAsBool)
return actionTaken
def callBackOnPreviousMove(self, state, action, reward, nextState, done):
self.remember(state, action, reward, nextState, done)
Example of output i have during fit method:
Epoch 1/10
1/1 [==============================] - 0s 0s/step - loss: 109.9612 - accuracy: 0.8867
Epoch 2/10
1/1 [==============================] - 0s 998us/step - loss: 109.9467 - accuracy: 0.8867
Epoch 3/10
1/1 [==============================] - 0s 0s/step - loss: 109.9456 - accuracy: 0.8867
Epoch 4/10
1/1 [==============================] - 0s 0s/step - loss: 109.9332 - accuracy: 0.8867
Epoch 5/10
1/1 [==============================] - 0s 998us/step - loss: 109.9339 - accuracy: 0.8867
Epoch 6/10
1/1 [==============================] - 0s 0s/step - loss: 109.9337 - accuracy: 0.8867
Epoch 7/10
1/1 [==============================] - 0s 997us/step - loss: 109.9305 - accuracy: 0.8867
Epoch 8/10
1/1 [==============================] - 0s 0s/step - loss: 109.9314 - accuracy: 0.8867
Epoch 9/10
1/1 [==============================] - 0s 0s/step - loss: 109.9306 - accuracy: 0.8867
Epoch 10/10
1/1 [==============================] - 0s 0s/step - loss: 109.9301 - accuracy: 0.8867
My questions are:
Is my architecture good (inputs = [0,0,1,1,0,0,1,0.....,1,0] (112x1 shape) to represent the state, and
output = [0,0,0,0,0,0,0,0,0,1,0,0,0...0,0,0,0] (112x1 shape with only one '1') ) to represent an action ?How to nicely choose the architecture of the Neural Network model (self._model) (I have only the basics of Neural Network, so I don't really know all activation fonction, how to design the hiden layers, choose a loss...)
To train my NN, is it good to call the 'fit' function with (state, action) as parameter to make it learn?
Is there something really important I forget in my design to make it work?