1
$\begingroup$

In a general DQN framework, if I have an idea of some actions being better than some other actions, is it possible to make the agent select the better actions more often ?

$\endgroup$
2
  • 1
    $\begingroup$ you should construct your reward function so that it will reflect the main goal and the algorithm will learn the best action by itself $\endgroup$ Nov 15 '20 at 13:41
  • $\begingroup$ take a look at my answers in similar questions: this one, and one more $\endgroup$ Nov 15 '20 at 13:47
0
$\begingroup$

For single-step Q learning, the behaviour policy can be any stochastic policy without any further adjustment to the update rules.

You don't have to use $\epsilon$-greedy based on current Q function approximation, although that is a common choice because it works well in general cases. However, you should always allow some chance of taking all actions if you want the algorithm to converge - if you fixed things so that bad actions were never taken, the agent would never learn that they had low value.

Probably the simplest way to use your initial idea of best actions is to write a function that returns your assessment of which action to take, and use that with some probability in preference to a completely random choice. At some point you will also want to stop referencing the helper function (unles it is guaranteed perfect) and use some form of standard $\epsilon$-greedy based on current Q values.

I have done similar with a DQN learning to play Connect 4, where the agent would use a look-ahead search function that could see e.g. 7 steps ahead. If that was inconclusive it would use argmax of current Q values. Both these fixed action choices could be replaced, with probability $\epsilon$, with a random action choice to ensure exploration. It worked very well. You could replace the look-ahead search in my example with any function that returned "best" actions for any reason.

There are some other ways you can skew action selection towards better looking action choices. You could look into Boltzmann exploration or upper confidence bounds (UCB) as other ways to create behaviour policies for DQN.

$\endgroup$
2
  • $\begingroup$ The connect 4 example looks to be a very nice approach to implement what I need. When you implement the look ahead search, do you mean look ahead in time on what might be the consequence of the actions ? Doesn't that mean you're still somewhat doing an argmax of the actions ? $\endgroup$ Nov 15 '20 at 15:36
  • $\begingroup$ @user3656142: My search chose by win/loss only, ignoring Q values. So it was an independent function. Yes it was an argmax over that independent function. The function was technically a negamax ( en.wikipedia.org/wiki/Negamax ) based on the game rules. $\endgroup$ Nov 15 '20 at 19:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.