1
$\begingroup$

I have implemented multiple MCTS based AI players for the Love Letter game (rules). It is a 2-4 players zero sum card game where players make alternating moves. I am struggling with how to properly conduct experiments for estimating AI player strength against human players:

  1. In 2 player game where one of the players is AI bot
  2. In 4 player game where one (or multiple) of players is AI bot
| improve this question | |
$\endgroup$
  • $\begingroup$ Do you have data (preferably lots) regarding human play choices? Do you have some humans available to help testing? $\endgroup$ – Neil Slater Mar 25 '18 at 8:36
  • $\begingroup$ @NeilSlater I don't have human play data. However, I have some humans availlable to help me in testing. Is it good approach to take 15 players (including 1 AI bot) with initial Elo rating and let them play against each other? $\endgroup$ – tamirok Mar 25 '18 at 21:19
  • $\begingroup$ If you are only concerned with win/loss outcomes, you can use the ELO system. $\endgroup$ – k.c. sayz 'k.c sayz' Mar 26 '18 at 18:50
2
$\begingroup$

The following are extremely simple ways of tackling this problem.

A very simple way

It can simply be
strength of AI=(# of games won)/(total # of games).


In case data for each move is available

Something like
score per game=# of correct decisions/total number of decisions.
Then
strength of AI=sum(score per game)/total # of games.


If each move/decision has a score associated with it

then you do
score per move=scored points by taking a decision/maximum possible score.
then
score per game=sum(score per move)/total # of moves
and finally,
strength of AI=sum(score per game)/total # of games.


How to choose optimal number of games to play?

It depends on your requirements. If you want to report your AI's strength in percentage of the games it played correct to 1 decimal place (for example, this AI won 95.1%) then 10000 is an optimal number of games your AI needs to play. Suppose your AI won 9508 games out of 10000 then you will have 95.08% strength of AI. To be able to correctly round it to 1 decimal place you need to have an additional decimal place so that you can quote the strength of you AI with reasonable confidence, in this case 95.1%.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Can you explain how to select optimal number of total games and how many people to engage for the experiment? $\endgroup$ – tamirok Mar 25 '18 at 21:23
  • $\begingroup$ I have tried to explain the optimal number of total games in the edit of the answer above. $\endgroup$ – Mohsin Bukhari Mar 26 '18 at 5:55
  • $\begingroup$ I am not sure how you can determine the optimal number of people to choose. $\endgroup$ – Mohsin Bukhari Mar 26 '18 at 5:55
  • $\begingroup$ Isn't this the relative strength of AI to a particular player? $\endgroup$ – DuttaA Mar 26 '18 at 6:16
  • $\begingroup$ @DuttaA That's true, most probably. It very well looks like it as well. $\endgroup$ – Mohsin Bukhari Mar 26 '18 at 17:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.