2
$\begingroup$

More precisely: is DQNN applicable only when we have high translational invariance in our input(s)?


Starting from the original paper on nature (here a version stored on googleapis) and after looking online for some other implementation and based on the fact that this NN starts with convolutional layers, I think that is based on the assumption that we feed the network with images but I'm not so sure. In the case that DQNN can be used with other types of inputs, please feel free to include examples in your answer. Also, references will be appreciated.

$\endgroup$
4
$\begingroup$

More precisely: is DQNN applicable only when we have high translational invariance in our input(s)?

No, DQN (that is the commonly-used abbreviation by the way, there also is a dqn tag which you may wish to use) is not restricted to images or other kinds of inputs with those properties, it can be used with pretty much any kinds of inputs.

The DQN algorithm should be viewed separately from the Neural Network Architecture though. DQN can be used with any kind of Neural Network architecture. Yes, the most commonly-used type of architecture used with DQN is probably architectures that start with a bunch of convolutional layers, and those kinds of architectures are best suited for image-based inputs. This is not a requirement though. If you have other kinds of features where convolutional layers don't make a lot of sense, you can, for example, simply start directly with some ReLU layers.

Here are two examples of DQN being used for problems without image-based inputs from the OpenAI baselines repository:

In both cases, they're using deepq.models.mlp() to construct a relatively straightforward Multi-Layered Perceptron architecture, without any convolutional layers. In this example for Atari games, they do have image-based input and therefore also construct an architecture with convolutional layers using deepq.models.cnn_to_mlp().


Note that, if you have relatively straightforward features, it may often not be necessary to use Deep RL approaches; tabular RL approaches of RL with Linear Function Approximation may work just as well in cases where you already have good features. If you have image-based inputs, those kinds of approaches are much less likely to work well. The disadvantage of Deep RL approaches like DQN is that they tend to require much more experience / data than simpler approaches.

So, generally the interesting question isn't "can DQN (or another Deep RL approach) handle my inputs?", because the answer is probably yes. The more important question would often be "do I have to use DQN (or another Deep RL approach?". The answer to that question will almost always be yes if you have image-based inputs, but relatively often be no if you already have good features as inputs.

Another class of problems where Deep RL is really popular nowadays is continuous control problems (e.g. robot simulators, MuJoCo, etc.). These don't have image-based inputs, but still generally require Deep RL (not DQN though; DQN doesn't handle continuous-valued outputs very well, it generates discrete outputs).

$\endgroup$
  • $\begingroup$ So is the DQN algorithm basically only about the Q action value-function and the loss function both based on the reward? $\endgroup$ – gvgramazio Jul 13 '18 at 17:19
  • 2
    $\begingroup$ @gvgramazio Pretty much yeah. I'd roughly define DQN as a neural network (architecture not specified other than that it has one output node per action) + Experience Replay Buffer + the loss function from the DQN paper + the idea of a Target Network $\endgroup$ – Dennis Soemers Jul 13 '18 at 17:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.