1
$\begingroup$

Let's say we have a dynamic environment: a new state gets added after 2000 episodes have been done. So, we leave room for exploration, so that it can discover the new state.

When it gets to that new state, it has no idea of the Q values, and, since we're past 2000 episodes, our exploration rate is very low. What happens if try to exploit when all Q values are 0?

Improve this question
$\endgroup$
0
$\begingroup$

There are several ways to tackle this, although exploration is definitely not a solved problem yet ;)

In general, I believe the right thing to do here is to measure the uncertainty of your policy or Q-value estimates and use that to construct some sort of exploration bonus. An intuitive example is given in Exploration by Random Network Distillation. They make two randomly initialized neural nets, one of which is never updated. At every transition, they feed transition data through both neural nets and use the difference in output between them as an estimate of uncertainty, and this quantity is added to the reward. Then they update the modifiable neural net towards the other one. This way, on a completely novel transition, the two neural nets will likely have very different outputs so the reward will be augmented a lot. Of course, this will hopefully encourage the agent to explore.

Improve this answer
$\endgroup$
6
  • $\begingroup$ Can you break this down please 😂, it’s quite daunting $\endgroup$ – Chukwudi Jun 25 '20 at 14:13
  • $\begingroup$ You have one fixed NN and another NN, both initialized randomly. Let's say these networks take states as inputs and outputs scalars. On a new state, both networks will output random scalars x and y that are likely different. Add |x-y| to the reward. Train the first network to minimize (x-y)^2. Once you've seen a state many times, the first network and the fixed network should match, so |x-y| ~= 0, so there's very little bonus. $\endgroup$ – harwiltz Jun 25 '20 at 17:31
  • $\begingroup$ Ohh this is deep Q learning, using target networks and our normal Q network $\endgroup$ – Chukwudi Jun 25 '20 at 21:48
  • $\begingroup$ Where you have a fixed target network which improves stability for our gradient descent and when it finall converges, we train our fixed networks with the updated weights of the Q network, am I right $\endgroup$ – Chukwudi Jun 25 '20 at 21:52
  • $\begingroup$ No, the two networks I'm talking about are not Q networks. I recommend you read the paper that I linked. $\endgroup$ – harwiltz Jun 25 '20 at 22:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.