1
$\begingroup$

I am playing with a deep Q-learning algorithm in my own environment. The network can perform well as long as there is only one enemy. My agent can perform the following actions:

  1. do_nothing
  2. prepare_for(e)
  3. attack(e)

where e is some enemy.

In the case of two enemies, the action vector has 5 elements:

|   0       |      1          |      2      |        3         |     4      |
-----------------------------------------------------------------------------
|do_nothing | prepare_for(e1) |  attack(e1) |  prepare_for(e2) | attack(e2) |
-----------------------------------------------------------------------------

After a couple of episodes, the agent always starts picking the first do_nothing action, which is not desired. Changing reward for do_nothing action is not helping, even using significantly higher negative reward, than for other actions.

There is no problem with the environment with only one enemy. (Only using columns 0, 1, 2). I feel like my action encoding can be the issue, but I can't figure it out, how to fix it. Any suggestions?

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ You have to provide a bit more details. What is the shape of your neural network? What are you using as input, state features or images? Are you providing state observation + action as the input or only state observation? How does the output look like, are there 5 nodes each for specific action or is it only a single output node for the specific action you provided as a part of the input. What are the rewards for your environment? Are you using exploratory policy? Are you using experience replay? Are you using separate networks? Are you providing enough training time? $\endgroup$
    – Brale
    Commented Feb 10, 2019 at 10:59

1 Answer 1

Reset to default
1
$\begingroup$

1) Vary the number of enemies during the training process. 2) Use other generalisation methods that are appropriate for neural networks such as dropout or L1 and L2 regularisation.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ Thank you for really fast responses. I have been stuck on this issue for the last 3 days and now, out of nowhere, it started working. Combination of two things has been critical to making it work: slightly more neurons in the hidden layer + significantly higher learning rate. Changing the learning rate from 0.0001 to 0.01 made it work. $\endgroup$
    – Savco
    Commented Feb 10, 2019 at 11:45
  • $\begingroup$ @Savco Thanks for the update, it is always useful to know which solutions people have found. $\endgroup$
    – DrMcCleod
    Commented Feb 10, 2019 at 18:27

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .