Why are shallow networks so prevalent in RL?

Question

In deep learning, using more layers in a neural network adds the capacity to capture more features. In most RL papers, their experiments use a 2 layer neural network. Learning to Reset, Constrained Policy Optimization, Model-based RL with stability guarantees just to name a few - these are papers I personally remember but there are definitely many others.

I came across this question whose answers generally agree that yes, RL using a shallow network is considered deep RL, but the reason for preference of shallow networks was not part of the question.

In MetaMimic (2018), the authors trained the largest neural net RL algorithm at the time (a residual net with 20 convolution layers) for one-shot imitation learning. The paper demonstrates that larger networks as policy approximators generalize better and represent many behaviours.

So, why are shallow 2 layer networks so widely used?

Answer 1

Simple policy parametizations, including linear functions in some cases, can solve continous control tasks in RL. It's therefore not necessary to have a complex approximator for the function to be expressive enough in capturing the desired agent's behaviour in popular RL benchmarks

---

Towards Generalization and Simplicity in Continous Control tries to answer the question "What are the simplest set of ingredients needed to succeed in some of the popular benchmarks?"

Among them is the representation capacity required of the function approximators.

It shows that in popular OpenAI gym locomotion benchmarks (Swimmer 3D, 3D Humanoid, Ant, Walker), using Linear and RBF policy architectures is just as competitive in the reward attained as using neural nets. Here's a summary:

• The more Fourier Featurers RBF has, the better the performance
• A 2 layer neural net policy's performance is slightly short of a 500-feature RBF
• A linear representation is unable to learn some of the locomotion behaviour, e.g walking

I assume RBF, and its use of Conjugate Gradients, has some shortcomings which make shallow Neural networks more preferrable. (I'm not familiar with RBF, and stand to be corrected).

Overall, the the authors use these result to suggest that complex policy architectures should not be a default choice in RL unless side-by-side comparisons with simpler alternatives suggest otherwise.

This might explain the prevalence of shallow networks. It's also possible that since most RL research (presently) doesn't try to show how we can learn better representations using complex policy architectures, but rather how to better make use of representation capacities by addressing issues like sample efficiency, shallow networks are expressive enough in the adapted benchmarks. As expressed in Brail's comment, this is also useful for setting up fair comparisons.

However, the effect of network architecture design in RL has not been widely pursued.

It's also worth mentioning the findings of this work D2RL, (2020) which shows one can gain significant performance improvement by using input concatenations in the policy and value function, and that more layers in a vanilla neural net decreases the agent performance.

The effect of increasing the number of FCs in SAC. Performance drops after 2 layers (D2RL)

Further Reads:

Why is everyone using shallow networks (Reddit thread)