What is the advantage of RL compared with my simple classic algorithm for the MountainCarEnv?

Question

What is the advantage of RL compared with the following simple classic algorithm for the MountainCarEnv? Considering that it takes a long time to train the agent just to achieve this simple task?

import gym

envName = 'MountainCar-v0'
env = gym.make(envName)

x, v = state = env.reset()
done = False
maxPotential = False
steps = 0

def computeAction(state):
    global x, v, maxPotential, steps
    xNew, vNew = state
    action = 1
    if xNew < -1.1:
        maxPotential = True
    if not maxPotential:
        if xNew < x: action = 0
        else: action = 2
    else:
        action = 2
    x, v = xNew, vNew
    steps += 1
    return action

while not done:
    state, reward, done, info = env.step(computeAction(state))
    env.render()

print('steps', steps)

• result: around 100 steps

Answer 1

If your goal is to create a controller for the mountain car problem, and you have access to the model, then RL probably offers no advantage over your code. I am saying probably, because I am taking you at your word that the code performs well over multiple tests, and it doesn't matter too much if it does not because there are many equivalent solutions based on analysis of the original problem.

This is the difference:

• RL techniques find solutions to control problems. Model-free RL techniques can do so without access to the environment model.

• The classic control code is a solution to a given control problem, found by the code author through analysis of the problem.

The same RL agent that could solve mountain car, could solve similar environments with same state and action space, e.g. an environment similar to mountain car but with multiple hills and valleys, or with alterations to the physics model. The same classic control code would fail and need to be re-written for the new environment.

The mountain car problem is interesting in control theory because it introduces a level of abstraction - the simplest feedback-based control algorithms will fail because moving directly towards the goal does not work. However, it is still a toy problem. The solutions are well understood, and no-one needs to solve it again. Solving it with RL is not necessary, it is a demonstration of learning something through trial and error.

As control problems become more complex, with multiple levels of goals to solve, then classic control approaches become more unweidly. For example, a walking robot has many more variables to manage, and walking systems are more likely to benefit from automated search for the best controllers as opposed to analysis and classic control at all levels.