How can I teach a computer to play N64 games using Neural Nets?

Question

I would like to work on a project where I teach an NN to play N64 games. To my current understanding, I would need an emulator?

I can do the Machine Learning side of it, im just unsure how I can give the NN access to the game's controls, such as left, right, up or down?

Where could I find more information on doing so and is using an Emulator the right path to take?

Answer 1

The common options are:

1. Implement a simple simulation of the game, perhaps without graphics. Train the agent on the simulation. This is usually not very satisfying, because the agent won't play the real game.
2. Use an emulator that allows you to inspect the game's memory, and input values directly at the controls. BizHawk is a good choice for SNES and I think also for N64 emulation. Use a pipe to connect this program with your agent. Pass across the pipe any key values from the game's memory that represent the state of the game, and send back actions that represent the inputs the agent makes in that state. Usually this is the best option.
3. Scrape the screen in real time. Tools like SkyScraper will allow you to capture the raw pixels displayed in a window. You should then be able to pass these pixels to your agents via a pipe. You may be able to use tools like Selenium to pass values back from the agent to the game. This route is usually a lot more intensive than option 2, but is the most realistic. It may also require very good hardware to keep up with the game in real time.