I'm learning about Actor-Critic reinforcement learning algorithms. One source I encountered mentioned that Actor and Critic can either share one network (but use different output layers) or they can use two completely separate networks. In this video he mentions that using two separate networks works for simpler problems, such as Mountain Car. However, more complex problems like Lunar Lander works better with a shared network. Why is that? Could you explain what difference that choosing one design over another would that make?
One can expect the optimal high-level features required to choose the next action and to evaluate a state to be quite similar. Because of that, it is a reasonable idea to share the same network for both policy and value function – you are essentially parameter sharing the feature-extraction part of your neural network, and fine tuning the different heads of your network on the two different tasks: action choice and value prediction.
Using two vs one networks is mostly a question of sample efficiency: theoretically, in both case your AC algorithm should work. In practice however, it will usually be useful to have parameter sharing as the representations encouraged by one of the tasks might be highly useful for the other and vice-versa, enabling one task to cause the other task to get unstuck from local optima. Another reason why this might work better is simply because you do not have to learn the same (or at least similar) representations from scratch twice – leading to a more sample efficient training.
2$\begingroup$ There are counter-examples, a paper that I skimmed recently came to the opposite conclusion after comparing both approaches (shared and separate networks), that separate networks for actor and critic are beneficial more often than not. I had a quick search for that paper again - sorry I cannot find it. $\endgroup$ Dec 8, 2020 at 19:53