Often times I see the term deep reinforcement learning to refer to RL algorithms that use neural networks, regardless of whether or not the networks are deep.
For example, PPO (https://arxiv.org/pdf/1707.06347.pdf) is often considered a deep RL algorithm, but using a deep network is not really part of the algorithm. In fact, the example they report in the paper says that they used a network with only 2 layers.
This SIGGRAPH project "DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills" https://xbpeng.github.io/projects/DeepMimic/2018_TOG_DeepMimic.pdf has the name Deep in it and the title even says 'deep reinforcement learning', but if you read the paper, you'll see that their network uses only 2 layers.
"Learning to Walk via Deep Reinforcement Learning" https://arxiv.org/pdf/1812.11103.pdf by researchers from Google and Berkeley. Again, deep RL in the title, but if you read the paper, you'll see they used 2 hidden layers.
Another SIGGRAPH project with deep RL in the title: https://www.cc.gatech.edu/~aclegg3/projects/learning-dress-synthesizing.pdf And if you read it, surprise, 2 hidden layers.
"Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor" https://arxiv.org/pdf/1801.01290.pdf If you read Table 1 with the hyperparameters they used: 2 hidden layers.
Is it standard to just call deep RL to any RL algorithm that uses a neural net?