I am currently using Proximal Policy Optimization (PPO) to solve my RL task. However, after reading about Soft Actor-Critic (SAC) now I am unsure whether I should stick to PPO or switch to SAC. Moreover, from this post, it seems that much of the performance in the original PPO paper comes from code optimizations and not the novel clipped objective.
The main characteristics of my RL task are the following:
- The action space is discrete. SAC was originally designed for continuous action spaces but, if I'm not wrong, it can be adapted to discrete action spaces with no problem.
- I am trying to learn a policy for generating synthetic data (i.e., generating novel graphs), so diversity is key. For this reason, I want to learn a policy with as much entropy as possible (while still solving the task). Both PPO and SAC try to maximize the policy entropy.
- Obtaining trajectories to train the policy is very expensive. My algorithm spends much more time obtaining the trajectories than training the deep neural network of the policy. Here, I think SAC is the clear winner, as it is off-policy whereas PPO is on-policy. Still, PPO is supposed to be very sample-efficient.
Given my current needs, do you think it is worth it to switch to SAC instead of PPO?