0
$\begingroup$

I have a Gym env (env) for which I train with a model using the PPO algorithm with stable-baselines.

from stable_baselines3 import PPO
from stable_baselines3.ppo import MlpPolicy
model = PPO(MlpPolicy, env, verbose=0)
model.learn(total_timesteps=10_000,progress_bar=True)

obs,info = env.reset()
action_prediction=[]
for i in range(5000):
    action, _states = model.predict(obs, deterministic=True)
    action_prediction.append(action)
    obs, rewards, done1, done2, info = env.step(action)
    if done1:
        break

If I use the model.predict routine after training, I see that the model successfully masters the task. However, this is not the fastest solution. For the proposed actions, I see for example that the model does not use the full action parameter range it could use. For a test case, where I know the optimal solution, I see that the prediction needs more steps than needed by the optimal solution.

How can I foster the model during learning to fullfill the task with a minimal number of steps? Is there a better way than going down the limbo of optimizing the reward function to balance mission completion and run time by writing my own Environment?

$\endgroup$
1
  • $\begingroup$ What is the problem that you are trying to solve? Have you considered the possibility that RL might not be the best solution (especially deep RL)? Except for customizing the reward to penalize longer episodes, I see no other way to achieve this when using RL. $\endgroup$
    – pi-tau
    Aug 1, 2023 at 23:00

2 Answers 2

2
$\begingroup$

First, I would like to emphasize that soon as you introduce a function approximator, guarantees of learning the optimal policy are thrown out of the window, moreover if it's a non-convex one:

  • maybe you get stuck on a local minima
  • maybe you don't do enough exploration
  • maybe the reward function does not encode the temporal dependency that you want
  • maybe you have not trained your model enough

You can definitely change the reward function, and that will for sure drive somewhere else your agent in the solution space, as that's the supervision signal that you are giving to it.

You can just add a penalization for each step that the agent takes, thus to minimize the cumulative reward, it has to take few steps as possible... however, consider that the agent might trade off penalization with speed:

if you penalize me -1 for each step, and i can choose between swimming in the river or going on the bridge, even if you give me -50 for swimming in the river because that's not what you want, if that saves me more than 50 steps, I'll do it

$\endgroup$
1
  • $\begingroup$ Thank you very much for the explanation. I'm quite new to reinforcement learning... Are there any standard ways to train for problems with the aim of completing a task in the least time? $\endgroup$
    – P. Egli
    Aug 1, 2023 at 19:21
0
$\begingroup$

A quick recap:

  1. You train a reinforcement learning agent using PPO
  2. Your agent learns useful behavior, but this behavior is suboptimal
  3. You would like your agent to learn a behavior closer to optimal

Let me suggest four strategies how you can improve you agent's learning. You can either use those strategies separately or combined.

(1) From the evidence you provide (the model does not use the full action parameter range it could use), it seems that the neural network under the hood is not able to leverage the full action range. A useful thing you can do at this stage is to normalize the action space. For instance, assuming you have continuous action environment (code source),

from gym.wrappers import RescaleAction


env = RescaleAction(env, min_action=-1, max_action=1)

Now, the neural network will assume that the actions are between -1 and 1, which will likely improve the situation.

(2) Additionally, to enhance your agent training, you may want to normalize your observations and/or rewards. Here is how you can do it with stable baselines3 vec_env:

from stable_baselines3.common.vec_env import DummyVecEnv, VecNormalize

env = DummyVecEnv([lambda: env])

# Wrap the environment with actions normalization
env = VecNormalize(env, norm_obs=True, norm_reward=True, clip_obs=10., clip_reward=10., training=True)

model = PPO(MlpPolicy, env, verbose=0)

Note that during evaluation, no need to update the rolling average of observations/rewards. This means you will need to set training=False when evaluating agents.

Try those three normalizations (actions, observations, rewards) separately and then combined and see how does it improve the policy or not.

(3) If you are designing your own environment, it might be useful to provide the agent more visibility in terms of observation space. This is all about feature engineering to provide the agent more information about its environment. One example would be including previous observations and/or actions into current observations. Note that including too many observations can also increase number of training iterations.

The simplest way to use this approach is by creating a custom gym wrapper.

(4) One more thing to do is to find better hyperparameters (e.g. using Optuna). Beware, this approach is computationally expensive because for each hyperparameter set it will train a new agent. Given that PPO was created with intention to minimize hyper parameters tuning (I briefly explain the PPO algorithm here), hyperparameter search should be used as a last resort only, i.e. after you tried everything else.

Finally, I have to point out that as soon as neural networks enter the game (as in PPO), there is no guarantee to learn the optimal policy. In case if you solve a simple task, you may want to use simpler algorithms (ones that do not involve neural networks). Here is the book about reinforcement learning.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .