I am trying to implement an intelligent agent that can perform penetration testing within the nasim (link) environment, a network simulator.
I would like to try to use parametric mode for actions, and therefore I need an agent that can work with OpenAI gym's MultiDiscrete actions since nasim implements OpenAI gym. I have therefore chosen to use PPO's implementation of Stable Baselines3 (link). Everything seems to work when I run the agent on a small scenario (tiny scenario). Still, when I try to run it on a somewhat larger environment (tiny-small scenario) the agent after a few hundred thousand timesteps decides to run only the action that costs the least. Here is the code of the agent:
import gym from stable_baselines3 import PPO from stable_baselines3.common.env_util import make_vec_env import nasim if __name__ == "__main__": # Build the environment env = gym.make("nasim:TinySmall-v2") # Train the model model = PPO("MlpPolicy", env, verbose=1) model.learn(total_timesteps=500000) # Init the environment obs = env.reset() # Init variables total_reward = 0 number_of_steps = 0 done = False # Do the test while done != True: action, _states = model.predict(obs) number_of_steps = number_of_steps + 1 obs, rewards, done, info = env.step(action) print("Action: " + str(action) + ", reward: " + str(rewards)) total_reward += rewards print("Total reward: " + str(total_reward)) print(done) print("Steps: " + str(number_of_steps)) env.render(mode='readable')
I tried looking for the cause of the problem in the algorithm, including watching this explanation video (link)., but I could not identify the cause of the problem.
Step to reproduce the problem:
- Install stable-baselines3:
pip install stable-baselines3
- Install version 0.9.1 of nasim (version 0.10 is too recent to be compatible with the PPO implementation of stable-baselines 3):
pip install -Iv nasim==0.9.1
- Install the last required component:
sudo apt install python3-tk
- Save the agent code written above in a .py file and run it.