Creating a self learning Mario Kart game AI?

Question

I will be undertaking a project over the next year to create a self learning AI to play a racing game, currently the game will be Mario Kart 64.

I have a few questions which will hopefully help me get started:

1. What aspects of AI would be most applicable to creating a self learning game AI for a racing game (Q-Learning, NEAT etc)
2. Could a ANN or NEAT that has learned to play Mario Kart 64 be used to learn to play another racing game?
3. What books/material should i read up on to undertake this project?
4. What other considerations should i take throughout this project?

Thank you for your help!

Answer 1

What aspects of AI would be most applicable to creating a self learning game AI for a racing game (Q-Learning, NEAT etc)

In general, you are looking at a problem that involves sequential decision making, in a machine learning context.

If you are wanting to build an agent that can learn by receiving screen images, then NEAT cannot scale to that complexity directly. Although there might be clever combinations of deep learning and evolutionary algorithms you could apply, the most heavily explored and likely successful solution will be found in Deep Reinforcement Learning. Algorithms like DQN, A3C, A2C, PPO . . . there are dozens to consider, but all are based around agents using samples of experience to update functions that measure either the "value" of acting in a certain way (a policy) or estimate the best policy directly.

Could a ANN or NEAT that has learned to play Mario Kart 64 be used to learn to play another racing game?

Within limits, yes. You will have built a system that takes pixel inputs, and outputs controller messages. If you re-start training from scratch on any compatible N64 game (with same screen resolution and same controller outputs) there is a chance it could learn to play that new game well. As other driving games on the N64 are a subset of all games, and more similar to each other than, say, a scrolling shooter or adventure game, then an agent that can successfully learn one will likely learn another too.

It is unlikely that a Mario Kart agent will immediately be good at another game without re-training. The visual and control differences will probably be too much. You could try though. An interesting experiment would be to take your trained agent, or some part of it, and see if starting with that improves learning time on a new game. This is called Transfer Learning.

What books/material should i read up on to undertake this project?

You will need at least the first parts of an introduction to Reinforcement Learning. If your goal is to aim to just make the agent, then you can skip theory-heavy parts, but within limits the more theory you understand, the easier it will be to change code features towards getting something working. I can suggest the following:

• The book Sutton & Barto, Reinforcement Learning: An Introduction - this covers theory behind basic RL, and the online draft is free

• A course of lectures on Reinforcement Learning from UCL, by David Silver. The video lectures cover same topics as the Sutton & Barto book.

• The book Deep Reinforcement Learning Hands-On, which you will need to buy. As the title suggests, it covers more practical aspects of building RL systems. It uses Python and PyTorch. As RL has become more popular in recent years, there are plenty of other similar books - I have no reason to recommend this one over them, but I am reading it at the moment, and the quality is adequate.

What other considerations should i take throughout this project?

Before you get started, you should know this could be an ambitious project, requiring a lot of compute time using currently-available toolkits. You need to think ahead a little, as you will be faced with these decisions:

• Is it more important to you to make a working bot on this problem, or more important to understand the underlying theories so that you understand what is going on. You will need to do at least a little of both, but you can go strongly in either direction:
  • There are enough pre-built learning agents available on the internet that you might have a successful project by learning just enough to wire up a copy of the game, and then letting the learning agent do its job. Would this be enough for you, would you feel that you had solved your problem?
  • There is plenty of educational material about RL and how the maths behind it works. If you are interested in understanding that, and perhaps generating your own ideas for improvements to current algorithms, then you need to study that material harder. However, this may lead you away from your Mario Kart player goal, and you may never actually solve it, because far simpler problems that require less computing power are still actually interesting academically.

In addition, I can think of the following:

• Building your Mario Kart player is a long-term goal. You will need to start with simpler agents in order to understand the techniques that you hope to apply to Mario Kart.

• You may need more compute power to solve this game than you have available.

• You will need to solve the issue of automating control of a N64 emulator. There is at least an existing emulator Mupen64Plus - I do not know whether it will be adequate for you, but at least one person has attempted to wrap this for automated learning, in the gym-mupen64plus project.