How can I apply reinforcement learning to solve this asteroid game?

Question

Introduction

An attractive asteroid game was described in the paper Learning Policies for Embodied Virtual Agents through Demonstration (2017, Jonathan Dinerstein et al.):

In our first experiment, the virtual agent is a spaceship pilot, The pilot's task is to maneuver the spaceship through random asteroid fields

In theory, this game can be solved with reinforcement learning or, more specifically, with a support vector machine (SVM) and epsilon-regression scheme with a Gaussian kernel. But it seems that this task is harder than it looks like, as the authors of the same paper write

Although many powerful AI and machine learning techniques exist, it remains difficult to quickly create AI for embodied virtual agents.

it is quite challenging to achieve natural-looking behavior since these aesthetic goals must be integrated into the fitness function

Questions

I really want to understand how reinforcement learning works. I built a simple game to test this. There are squares falling from the sky and you have the arrow keys to escape.

How could I code the RL algorithm to solve this game? Can I do this manually in Javascript according to what I think should happen? How can I do this without having to map the positions of the rectangles and mine, just giving the agent the keyboard arrows to interact and three information:

• Player life
• Survival time
• Maximum survival time

Answer 1

A closely related question and a minimal implementation written in Python.

That program implements the reinforcement learning technique 'Q-Learning'.

The idea is for the program to take in an observation of the environment (which could be a screenshot if learning a computer game, or sensor data for a robot) and output a decision in the form of a vector of values. Each cell in that output vector corresponds to a possible action (left, right, shoot, etc) and the highest valued cell shows the action that the agent/player should take. The values in that output vector are called the Q-Values, and it is the mapping from the input data to the vector of Q-Values that we are trying to learn. In your case, the function that takes in an observation of the environment and spits out a vector of decision choices is your SVM.

The question linked above contains a description of the training algorithm. It boils down to playing the game multiple times while storing the input vector, output vector and output decision for each step until you reach a termination condition (ie hitting an asteroid) and receiving a score (in this case a negative score since you want to avoid the asteroids). Then, going backwards through the the output vectors, assigning a slowly reducing amount of the output score to the particular cell in each output vector corresponding to the decision that was made at that step of the game. The array of input vectors from the game, and the array of output decision vectors that you have just created become the training data for your machine learning system. Once you have completed a training run, play the game again with your trained system and keep repeating until it is a good as you need it to be, or it stops getting any better.

To address your question directly, I don't think that you are going to be able to train your system without giving it information about the position of the asteroids (assuming that they are positioned at random). However, you could try just having a very simple input vector such as 3 integer values where each value corresponds to the presence (1) or absence (0) of an asteroid in the 3 squares to the above left, above and above right of the player. That might be enough to encourage it to dodge away from falling rocks...

There are some Support Vector Machines implemented in JavaScript, although I have never used them, but since SVMs are basically two-class classifiers you would have to check that the library can easily support multi-class SVMs. In your case the SVM would be trying to classify each input into a particular player choice (left, right, shoot, etc).