Why do some DQN implementations not require random exploration but instead emulate all actions?

Question

I've found online some DQN algorithms that (in a problem with a continuous state space and few actions, let's say 2 or 3), at each time step, compute and store (in the memory used for updating) all the possible actions (so all the possible rewards). For example, on page 5 of the paper Deep Q-trading, they say

This means that we don't need a random exploration to sample an action as in many reinforcement learning tasks; instead we can emulate all the three actions to update the Q-network.

How can this be compatible with the exploration-exploitation dilemma, which states that you have to balance the time steps of exploring with the ones of exploiting?

Answer 1

The example that you linked is using a model (emulation) in order to look ahead at all possible actions from any state. It essentially explores off-policy and offline using that model. This is not an option that is available in all environments, but if possible it resolves the exploration/eploitation dilemma for a single time step nicely by investigating all options.

Longer term the agent proposed by the link does not sufficiently explore for general use in my opinion. It appears to always choose a single action deterministically based on maximising action value. In other words it always attempts to exploit the training data so far, even though it augments the training data with short-term knowledge about exploration. However, this appears to be sufficient in the problem domain that it is used in. I suspect this is for a couple of reasons:

• The environment is non-stationary, making long-term state predictions unreliable in any case. An agent that learns to exploit in the short term (i.e. over only a few time steps into the future) is likely to be approximately optimal already.

• State transitions may be highly stochastic, meaning that state space will still be adequately explored even using a deterministic policy. This feature of the environment is also used by other well-known Q learning approaches with deterministic behaviour policies, such as TD Gammon

I think you have correctly identified a weakness of the approach used in the linked paper that means it may not make a strong general algorithm. The algorithm avoids addressing the exploration/exploitation balance in full, and instead relies on features of the environment to work well despite this. If you find yourself working in similar environments for your own projects, then it may well be worth trying the same approach. However, if you find yourself working in a more deterministic environment with more stationary behaviour and sparse rewards, the lack of state space exploration would be a serious limitation.