4
$\begingroup$

Often times I see the term deep reinforcement learning to refer to RL algorithms that use neural networks, regardless of whether or not the networks are deep.

For example, PPO (https://arxiv.org/pdf/1707.06347.pdf) is often considered a deep RL algorithm, but using a deep network is not really part of the algorithm. In fact, the example they report in the paper says that they used a network with only 2 layers.

This SIGGRAPH project "DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills" https://xbpeng.github.io/projects/DeepMimic/2018_TOG_DeepMimic.pdf has the name Deep in it and the title even says 'deep reinforcement learning', but if you read the paper, you'll see that their network uses only 2 layers.

"Learning to Walk via Deep Reinforcement Learning" https://arxiv.org/pdf/1812.11103.pdf by researchers from Google and Berkeley. Again, deep RL in the title, but if you read the paper, you'll see they used 2 hidden layers.

Another SIGGRAPH project with deep RL in the title: https://www.cc.gatech.edu/~aclegg3/projects/learning-dress-synthesizing.pdf And if you read it, surprise, 2 hidden layers.

"Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor" https://arxiv.org/pdf/1801.01290.pdf If you read Table 1 with the hyperparameters they used: 2 hidden layers.

Is it standard to just call deep RL to any RL algorithm that uses a neural net?

$\endgroup$
3
$\begingroup$

A neural network with at least one hidden layer is often considered "deep", in the sense that it can approximate any "reasonable" function, given "enough" (but finite number of) units (or neurons) in the layers. See the universal approximation theorem.

However, have a look at this question https://stats.stackexchange.com/q/229619/82135, where a few people state that the adjective "deep" is not precisely defined in the context of deep learning.

$\endgroup$
  • $\begingroup$ that's very interesting, so every neural network is "deep" using the common terminology? can you point me to an article where a network with one hidden layer is explicitly referred to as deep network? $\endgroup$ – yewang Apr 1 at 4:43
1
$\begingroup$

Is it standard to just call deep RL to any RL algorithm that uses a neural net?

Yes, it seems to have become standard practice to label RL + any NN "Deep Reinforcement Learning". It is not a formalised term.

The whole "Deep Learning" movement started this decade is as much a marketing term as a scientific one. It is however based on the discovery of real improvements in neural network architecture and training approaches.

You may find that some (or even most) of these shallower networks will use improvements designed in the last decade or so, and also associated with deeper networks, such as Xavier initialization, ReLU activation, the Adam optimizer.

As a personal opinion, I would say that, if a published experiment uses just 1 or 2 hidden layers, and does not make use of any of these recent advances, then the "Deep" label is almost entirely a branding exercise. There were advances with such networks much longer ago. For instance the TD-Gammon paper is from 1995. For TD-Gammon, the authors used reinforcement learning and a NN with one hidden layer to create a Backgammon player that played better than any human player. This was well before "Deep Learning" was a term used to describe such networks, and the term "Deep Reinforcement Learning" does not appear in that paper.

However, because "Deep Learning" is such a loose branding term, there is also an argument that all these older approaches, and pretty much all neural networks with hidden layers, should be included. Wikipedia's definition for Deep Learning says:

Deep learning is a class of machine learning algorithms that:

  • use a cascade of multiple layers of nonlinear processing units for feature extraction and transformation. Each successive layer uses the output from the previous layer as input.
  • learn in supervised (e.g., classification) and/or unsupervised (e.g., pattern analysis) manners.
  • learn multiple levels of representations that correspond to different levels of abstraction; the levels form a hierarchy of concepts.

Using that definition would include all the papers you cite. You don't need a 50 layer Resnet architecture to qualify. And the branding exercise makes more sense under that definition, because the newly invented techniques have made such systems that much more viable and worthy of investment (of time & effort as well as financially).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.