I have implemented multiple MCTS based AI players for the Love Letter game (rules). It is a 2-4 players zero sum card game where players make alternating moves. I am struggling with how to properly conduct experiments for estimating AI player strength against human players:

  1. In 2 player game where one of the players is AI bot
  2. In 4 player game where one (or multiple) of players is AI bot
  • $\begingroup$ Do you have data (preferably lots) regarding human play choices? Do you have some humans available to help testing? $\endgroup$ Mar 25, 2018 at 8:36
  • $\begingroup$ @NeilSlater I don't have human play data. However, I have some humans availlable to help me in testing. Is it good approach to take 15 players (including 1 AI bot) with initial Elo rating and let them play against each other? $\endgroup$
    – tamirok
    Mar 25, 2018 at 21:19
  • $\begingroup$ If you are only concerned with win/loss outcomes, you can use the ELO system. $\endgroup$ Mar 26, 2018 at 18:50

1 Answer 1


The following are extremely simple ways of tackling this problem.

A very simple way

It can simply be
strength of AI=(# of games won)/(total # of games).

In case data for each move is available

Something like
score per game=# of correct decisions/total number of decisions.
strength of AI=sum(score per game)/total # of games.

If each move/decision has a score associated with it

then you do
score per move=scored points by taking a decision/maximum possible score.
score per game=sum(score per move)/total # of moves
and finally,
strength of AI=sum(score per game)/total # of games.

How to choose optimal number of games to play?

It depends on your requirements. If you want to report your AI's strength in percentage of the games it played correct to 1 decimal place (for example, this AI won 95.1%) then 10000 is an optimal number of games your AI needs to play. Suppose your AI won 9508 games out of 10000 then you will have 95.08% strength of AI. To be able to correctly round it to 1 decimal place you need to have an additional decimal place so that you can quote the strength of you AI with reasonable confidence, in this case 95.1%.

  • $\begingroup$ Can you explain how to select optimal number of total games and how many people to engage for the experiment? $\endgroup$
    – tamirok
    Mar 25, 2018 at 21:23
  • $\begingroup$ I have tried to explain the optimal number of total games in the edit of the answer above. $\endgroup$
    – Mohsin
    Mar 26, 2018 at 5:55
  • $\begingroup$ I am not sure how you can determine the optimal number of people to choose. $\endgroup$
    – Mohsin
    Mar 26, 2018 at 5:55
  • $\begingroup$ Isn't this the relative strength of AI to a particular player? $\endgroup$
    – user9947
    Mar 26, 2018 at 6:16
  • $\begingroup$ @DuttaA That's true, most probably. It very well looks like it as well. $\endgroup$
    – Mohsin
    Mar 26, 2018 at 17:36

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .