Practical differences between the different RL architectures

Question

I have tried many different RL architectures: DQN, PPO, Policy optimization, and for my specific problem they all failed in their basic setup. Eventually I discovered that my problem had too sparse/deceptive rewards and that they would never work. This made me wonder what the practical differences are between the architectures. In my view it looks like that the more recent architectures have better exploration/exploitation trade offs. And that the main problem with RL is not finding any good trajectories to learn from, but that all algorithms are pretty good at performing once it has found that good trajectory. So is the exploration/exploitation trade off the biggest difference on which most architectures differ on?

Answer 1

In theory on-policy algorithms like PPO and actor-critic should perform better than off-policy algorithms such as DQN. The downside is that you will need to collect new data once the policy is updated, while with DQN you have a replay buffer that reuses experiences.

PPO also allows you to make several updates before collecting new data thus increasing the sample efficiency of policy gradient methods. But still DQN is way more efficient. So if your environment is slow, not parallelizable, or you cant even simulate it and have to use the real thing, then maybe use DQN.

Another major difference between DQN and direct policy optimization methdos (like PPO) is that you cannot use DQN with continuous action spaces. There is just no way to compute the maximum over the action space in the equation $Q(s_t,a_t) = r_{t+1} + \gamma \max_{a \in \mathcal{A}}Q(s_{t+1},a)$. Policy gradient methods have no problem with this.

Regarding exploration, there are a lot of great works that research this topic both for policy gradient methods and for DQN-style off-policy methods. I can't really say which line of research has better results.

Finally, regarding your problem, I wanted to say that if you implement the algorithms yourself there might be a great chance that you implemented something incorrectly. Especially in the case of PPO, which is known to have a lot of details not mentioned in the official paper. Using a correct implementation is very important for hard problems. You might be able to solve atari pendulum, but it will fail on harder tasks. Here is a nice suggestion for testing your implementation on increasingly harder environments: CartPole, LunarLander, Pong, Breakout.

You can also check out these two blogposts and compare your implementation of PPO:

https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/

https://pi-tau.github.io/posts/actor-critic/