I cannot make SAC learn a task in a certain environment. The point is that it actually sometimes finds a very good policy, but it never learns the policy in the end. I am using the SAC implementation from stable-baselines3, which is correct as far as I have seen. I have an environment driven by complex dynamics. I have to control a variable to be in a certain range. Every time the variable goes out of minimum or maximum range the environment is done. The action is continuous (between 0 and 30). My goal is to keep the variable (1D) in the range for as long as possible (millions of steps per episode would be ideal). There are certain characteristics of the environment that may make it particular:
- The action can only drive the variable to lower values. The variable can go up as a result of the environment dynamics (not controlled) and as a consequence of certain events (not controlled) that occur at random intervals.
- The observation is a noisy sample of the variable. The observation is just a real number.
- The effect of actions in the variable is usually delayed. That is, applying an action does not immediately lower the value of the variable.
I have tried SAC with many different hyperparameters. It sometimes find very good policies, policies that last for thousands and even millions of steps in evaluation or rollout. But it never learns such policies. Even saving the policy in those cases, they are not able to produce a lone episode later. In the attached image, it can be seen that during the training (in some evaluations) the policy is able to run for thousand of steps. But then it never learns that. I only show 500K here steps but I have run test for 1.5 million training timesteps.
So, my question is (I have several ones actually):
- Is SAC not suitable for this problem? I have also run TD3 and PPO but without better results and SAC is the only one actually able to find those policies that make very long episodes. Any other algorithm?
- I have tried several reward functions, and, in the end, a simple one that gives 1 for every step and 0 when done is the one that seems to give better results. In the image, the reward is one for every step and -100 when done.
- Since the values of the variable are time correlated due to the dynamics, I have also tried with RNN actors (with TF Agents), but results do not improve.
- I cannot see any relationship between the actor loss and critic loss and the results (maybe that is my problem). The loss seem to be larger when the episodes are longer (which is what I want).
Any advice is highly appreciated. Thanks