Most language model is using online PPO or offline DPO type algorithm. Can we use soft actor critic RL to do alignment work? Any publication related can be recommended?
2
-
$\begingroup$ tokens are discrete, SAC is for continuous control... you do you $\endgroup$– AlbertoCommented Aug 9 at 21:27
-
$\begingroup$ In principle you can use Discrete SAC, Rainbow, IQN, etc, all of which attain performance comparable to PPO (at least in common RL benchmarks) - probably, PPO is favored because it is designed to be parallelizable over many nodes $\endgroup$– Luca AnzaloneCommented Aug 11 at 10:02
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
Sample efficient off-policy SAC learning is originally proposed for robotics continuous action space RL use cases while its alternative version could handle discrete action spaces.
A central feature of SAC is entropy regularization. The policy is trained to maximize a trade-off between expected return and entropy, a measure of randomness in the policy. This has a close connection to the exploration-exploitation trade-off: increasing entropy results in more exploration, which can accelerate learning later on... An alternate version of SAC, which slightly changes the policy update rule, can be implemented to handle discrete action spaces.
So the reasons why it's not used in LLM RLHF (RL with human feedback) NLP tasks are related to its main purpose which is to deal with continuous action spaces with limited samples due to its inherent entropy maximization explorative nature. LLM RLHF stage doesn't lack human provided ranking samples to inversely learn the reward model (RM), and the extremely stable clipped-objective oriented PPO on-policy learning is well suited to make sure a stable and robust learning process which is apparently desirable in RLHF after supervised fine-tuning (SFT) stage.
Incidentally to speedup RLHF learning Direct Preference Optimization (DPO), Contrastive Preference Learning (CPL), and rejection sampling, etc have been proposed to replace or enhance PPO. Meta's Llama-2-chat models first use rejection sampling to fine tune with samples of higher reward compared to PPO which updates based on a single sample each time, PPO is used only at its final stage. You can read below references for these alternatives.
Bai et al. (2022). "Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback"
Hejna et al. (2023). "Contrastive Preference Learning: Learning from Human Feedback without RL"
Rafailov et al. (2023). "Direct Preference Optimization: Your Language Model is Secretly a Reward Model"
