Why does a PPO agent perform only the action that costs the least?

Question

I am trying to implement an intelligent agent that can perform penetration testing within the nasim (link) environment, a network simulator.

I would like to try to use parametric mode for actions, and therefore I need an agent that can work with OpenAI gym's MultiDiscrete actions since nasim implements OpenAI gym. I have therefore chosen to use PPO's implementation of Stable Baselines3 (link). Everything seems to work when I run the agent on a small scenario (tiny scenario). Still, when I try to run it on a somewhat larger environment (tiny-small scenario) the agent after a few hundred thousand timesteps decides to run only the action that costs the least. Here is the code of the agent:

import gym
from stable_baselines3 import PPO
from stable_baselines3.common.env_util import make_vec_env
import nasim

if __name__ == "__main__":

    # Build the environment
    env = gym.make("nasim:TinySmall-v2")

    # Train the model
    model = PPO("MlpPolicy", env, verbose=1)
    model.learn(total_timesteps=500000)

    # Init the environment
    obs = env.reset()

    # Init variables
    total_reward = 0
    number_of_steps = 0
    done = False
    
    # Do the test
    while done != True:
        action, _states = model.predict(obs)
        number_of_steps = number_of_steps + 1
        obs, rewards, done, info = env.step(action)
        print("Action: " + str(action) + ", reward: " + str(rewards))
        total_reward += rewards
        print("Total reward: " + str(total_reward))
        print(done)
        print("Steps: " + str(number_of_steps))
        env.render(mode='readable')

I tried looking for the cause of the problem in the algorithm, including watching this explanation video (link)., but I could not identify the cause of the problem.

Step to reproduce the problem:

1. Install stable-baselines3: pip install stable-baselines3
2. Install version 0.9.1 of nasim (version 0.10 is too recent to be compatible with the PPO implementation of stable-baselines 3): pip install -Iv nasim==0.9.1
3. Install the last required component: sudo apt install python3-tk
4. Save the agent code written above in a .py file and run it.

Answer 1

The agent has probably learned that it's not worthwhile to explore further, and greedily minimizes cost by taking the least expensive action. I haven't looked at the environment, but one simple test could be to increase the entropy bonus of PPO, and see if the policy explores more (you can keep track of the policy's entropy to see if this is indeed the issue).