Is reinforcement learning using shallow neural networks still deep reinforcement learning?

Question

Often times I see the term deep reinforcement learning to refer to RL algorithms that use neural networks, regardless of whether or not the networks are deep.

For example, PPO is often considered a deep RL algorithm, but using a deep network is not really part of the algorithm. In fact, the example they report in the paper says that they used a network with only 2 layers.

This SIGGRAPH project (DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills) has the name deep in it and the title even says 'deep reinforcement learning', but if you read the paper, you'll see that their network uses only 2 layers.

Again, the paper Learning to Walk via Deep Reinforcement Learning by researchers from Google and Berkeley, contains deep RL in the title, but if you read the paper, you'll see they used 2 hidden layers.

Another SIGGRAPH project with deep RL in the title. And, if you read it, surprise, 2 hidden layers.

In the paper Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor, if you read table 1 with the hyperparameters, they also used 2 hidden layers.

Is it standard to just call deep RL to any RL algorithm that uses a neural net?

Answer 1

Even after several years of success of deep learning systems (i.e. neural networks trained with gradient descent and back-propagation), as far as I know, there is not yet a consensus on what constitutes a deep neural network. Some people could use a neural network with 2 hidden layers and call it deep (like in your case), but other people may just dedicate the adjective deep to refer to neural networks with 10, 100, or more hidden layers. In fact, there are some good reasons to associate the term deep only to neural networks that have a significant number of hidden layers (e.g. 100): for example, the exploding (or vanishing) gradient problem does not typically arise if you only have one hidden layer but can easily occur with many (e.g. 100) hidden layers.

Nevertheless, a neural network with at least one hidden layer can approximate any continuous function, given enough (but finite number of) units (or neurons) in the layers. See the universal approximation theorem. For this reason, we could start denoting any such neural network as deep, but, although this rule would exclude perceptrons (which can only approximate linear functions, and nobody would probably call them deep anyway), this rule would be a bit redundant or useless (i.e. we may just not use the adjective deep to start with).

In your case, the rule that the authors are using seems to be the following: if it contains more hidden layers than the bare minimum (i.e. 1) to approximate any continuous function, then let's denote it as deep.

Answer 2

Is it standard to just call deep RL to any RL algorithm that uses a neural net?

Yes, it seems to have become standard practice to label RL + any NN "Deep Reinforcement Learning". It is not a formalised term.

The whole "Deep Learning" movement started this decade is as much a marketing term as a scientific one. It is however based on the discovery of real improvements in neural network architecture and training approaches.

You may find that some (or even most) of these shallower networks will use improvements designed in the last decade or so, and also associated with deeper networks, such as Xavier initialization, ReLU activation, the Adam optimizer.

As a personal opinion, I would say that, if a published experiment uses just 1 or 2 hidden layers, and does not make use of any of these recent advances, then the "Deep" label is almost entirely a branding exercise. There were advances with such networks much longer ago. For instance the TD-Gammon paper is from 1995. For TD-Gammon, the authors used reinforcement learning and a NN with one hidden layer to create a Backgammon player that played better than any human player. This was well before "Deep Learning" was a term used to describe such networks, and the term "Deep Reinforcement Learning" does not appear in that paper.

However, because "Deep Learning" is such a loose branding term, there is also an argument that all these older approaches, and pretty much all neural networks with hidden layers, should be included. Wikipedia's definition for Deep Learning says:

Deep learning is a class of machine learning algorithms that:

• use a cascade of multiple layers of nonlinear processing units for feature extraction and transformation. Each successive layer uses the output from the previous layer as input.
• learn in supervised (e.g., classification) and/or unsupervised (e.g., pattern analysis) manners.
• learn multiple levels of representations that correspond to different levels of abstraction; the levels form a hierarchy of concepts.

Using that definition would include all the papers you cite. You don't need a 50 layer Resnet architecture to qualify. And the branding exercise makes more sense under that definition, because the newly invented techniques have made such systems that much more viable and worthy of investment (of time & effort as well as financially).