A lot of research has been done to create the optimal (or "smartest") RL agent, using methods such as A2C. An agent can now beat humans at playing Go, Chess, Poker, Atari Games, DOTA, etc. But I think these kind of agents will never be a friend of humans, because humans won't play with a agent that always beats them.

How could we create an agent that doesn't outperform humans, but it has the human level skill, so that when it plays agains a human, the human is still motivated to beat it?


You basically have to degrade the result, assuming that the machine always finds the best move. There are a number of possibilities:

  • restrict the depth of searching. In early chess programs I believe that was the main way of regulating the difficulty. You stop the evaluation of moves after a particular depth in your search tree has been reached. This would be equivalent to only looking ahead two moves instead of twenty.

  • set a time limit. This is somewhat similar the restricting the depth of the search, but more generally applicable. If your algorithm accumulates candidate moves, and the general tendency is to get to the better moves after first finding a number of weaker ones, then you can stop at a given point in time and return what you have found then.

  • distort available information. This might not be that applicable to games such a chess, but you could restrict the information the machine has available for evaluating moves. Something like the "Fog of War" often used in strategy games. With incomplete information it is harder to find a good move, though it is not impossible, which makes it more challenging than, say, restricting the depth of search too much.

  • sub-optimal evaluation function. If you have a function that evaluates the quality of a move, simply fudge that function to not return the best value. Perhaps add a random offset to the return value to make it less deterministic/predictable.

There are probably other methods as well; the tricky part is to tread the fine line between appearing to be a weaker (but consistent) player, and just being a random number generator.

  • $\begingroup$ I think you can add the main problem detail in the "tricky" part: There is a problem of measuring capability against humans. It is not easy to automate, because if you can automate a human-level opponent to test with then you have already solved your problem! So it is a slow process of testing the bot with human subjects to see if it is enjoyable to play against. $\endgroup$ – Neil Slater Feb 7 at 10:27
  • $\begingroup$ @NeilSlater Yes, agree 100%. Enjoyment is hard to quantify... $\endgroup$ – Oliver Mason Feb 7 at 13:42
  • $\begingroup$ Great answer! I like the concept of "Fog of war" and also to add the time limit, as human also experience the same problem, thank you! $\endgroup$ – malioboro Feb 8 at 7:32

A possible way to improve the user experience in games is to create intelligent tutoring systems. That are serious games which have an educational approach. The Artificial Intelligence in that games is controlling the non-player-characters but with the idea to teach the human. A typical example is a simulated driver school in which the human learns everything about correct behavior in traffic situations.

Intelligent tutoring systems are organized in lessons. That means it's an interactive story in which the human has to show, that he has understood the instructor. Each lesson ends with a quiz which is often practical. That means, the human is cruising in the city, some autonomous bots are driving also in the street and the human must pass the test. That means, he must stop at the red light and has to be limit his maximum speed. In the background, an artificial instructor is observing the human and he give tips how to improve the driving.

From a technical perspective such educational enriched AI-driven games are created with normal game engines like Unity plus some extra features for modeling the knowledge in the game. Often the protege ontology editor is in charge for building the lessons.

  • $\begingroup$ By "protoge ontology editor" did you mean this: dcc.ac.uk/resources/external/protege-ontology-editor ? $\endgroup$ – Neil Slater Feb 7 at 10:28
  • $\begingroup$ Ontologies are a tool for storing domain knowledge. In the OWL syntax a graph of semantic information about a domain is created. In case of a serious game for a driving school this would be: traffic signs, car models, allowed maximum speed, street information and lane width. An ontology editor like Protege is able to collect all these information. There is a similarity to a class diagram but with a focus on expert knowledge. $\endgroup$ – Manuel Rodriguez Feb 7 at 10:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.