0
$\begingroup$

First of all I know that: 'it makes training less stable' & 'RL is already inherently unstable'. I'm asking why those things are true?

Intuitively it seems very strange & to be perhaps a fundamental weakness of RL that it very ineffective at learning large networks. Supervised learning usually has no problem learning features from unstructured data via potentially quite large networks (e.g. upwards of 8+ layers). But in the case of RL it is a very big problem it seems (that is learning the extra embedding layers in the front of the network 'head').

P.S. in case you don't believe me here is a plot of hyper-parameter search data I collected on Acrobot-v1, CartPole, MountainCar-v0, and Pendulum-v0. Also it is not hard to find evidence for this online... Just look at stable-baselines3 it has default num_layers=2...

enter image description here

Improve this question
$\endgroup$
3
  • $\begingroup$ I know that it might be true, but who said that "training large networks in RL is usually (more) unstable"? It might be a good idea to quote someone or something that claimed this, or maybe tell us that you've observed this empirically. $\endgroup$
    – nbro
    Jun 18 at 17:08
  • $\begingroup$ Sometimes the initial layers of the model will approximate the environment better. So adding layers means adding more noise and hence the learning becomes unstable. Every neural network has certain optimal number of layers and depth depending upon the use case. Too many layers in a neural network leads to bad learning and it is not limited only to RL. One of the areas I see this more frequently is in image classification where in some cases, few layers perform better than large number of layers $\endgroup$
    – MonkeyDLuffy
    Jun 23 at 21:36
  • $\begingroup$ @MonkeyDLuffy I appreciate your answer but this is definitely something special about RL. The 'standard' number of layers is 2! Compared to 8 for even a pretty standard feedforward network, something along these lines is common in computer vision/NLP too. In RL that leads to learning collapse. $\endgroup$
    – profPlum
    Jun 24 at 21:03

0

Reset to default

You must log in to answer this question.

Browse other questions tagged .