I'm trying to figure out how PPO ever learns anything in a sparse environment like gymnasium's MountainCar-v0 before it first ever reaches the goal state.

Specifically was looking at stable_baselines3's implementation of PPO

env = make_vec_env('MountainCar-v0', n_envs=16)
model = PPO('MlpPolicy', env, verbose=1, learning_rate=1e-3,
            gamma=0.99, gae_lambda=0.98, ent_coef=0.0, n_steps=16, normalize_advantage=True)

I ran different experiments and logged when the environment first reaches the goal state.

In the above setup, it usually first reaches a goal state in around 50-150k timesteps.

I ran a separate experiment where I just randomly choose actions at every step (so no "learning" is going on) and it basically never reaches the goal state (within the 200 step episode limit). The same holds true if the learning rate is set to 0 (mimicking just random actions), so it seems like some kind of learning is going on.

Also when the n_envs is set to just 1 or if normalize_advantage is turned off, it also basically never reaches the goal state.

I'm confused how PPO is learning anything before first reaching the goal state if every state it sees would give the same reward (of -1). I don't see any reward shaping in MountainCar-v0, and I don't see any Curiosity in the PPO implementation.

What am I missing?



You must log in to answer this question.