1
$\begingroup$

In the information theory, the entropy is a measure of uncertainty in some system. Being applied to agent policy, entropy shows how much the agent is uncertain about which action to make. In math notation, entropy of the policy is defined as : $$H(\pi) = -\sum \pi(a|s) \log \pi(a|s)$$ The value of entropy is always greater than zero and has a single maximum when the policy is uniform. In other words, all actions have the same probability. Entropy becomes minimal when our policy has 1 for some action and 0 for all others, which means that the agent is absolutely sure what to do. To prevent our agent from being stuck in the local minimum, we are subtracting the entropy from the loss function, punishing the agent for being too certain about the action to take.

The above excerpt is from Maxim Lapan in the book Deep Reinforcement Learning Hands-on page 254.

In code, it might look like :

 optimizer.zero_grad()
 logits= PG_network(batch_states_ts)
 log_prob = F.log_softmax(logits, dim=1)
 log_prob_actions = batch_scales_ts * log_prob[range(params["batch_size"]), batch_actions_ts]
 loss_policy = -log_prob_actions.mean()

 prob = F.softmax(logits, dim=1)
 entropy = -(prob * log_prob).sum(dim=1).mean()
 entropy_loss = params["entropy_beta"] * entropy
 loss = loss_policy - entropy_loss

I know that a disadvantage of using policy gradient is our agent can be stuck at a local minimum. Can you explain mathematically why subtracting the entropy from our policy will prevent our agent from being stuck in the local minimum ?

$\endgroup$
3
  • $\begingroup$ In other words, what author wanted to say is that entropy helps exploration. If you don't explore you might get suboptimal policy. Local minimum is an (overused) term to signify something suboptimal, in this case policy. Using entropy term will not prevent getting suboptimal policies but it might help getting better policies because you're exploring more. $\endgroup$
    – Brale
    Apr 25, 2020 at 14:49
  • $\begingroup$ @Brale_ It says "subtracting the entropy from our policy will prevent our agent from being stuck in the local minimum". So after the training we should always get the optimal policy or close. This is what I understand $\endgroup$
    – jgauth
    Apr 25, 2020 at 14:53
  • 1
    $\begingroup$ Well no, if it was that easy to always get optimal policy just by adding entropy reinforcement learning would be solved. You're taking it too literally, it just means you're helping exploration by adding entropy. $\endgroup$
    – Brale
    Apr 25, 2020 at 15:13

0

You must log in to answer this question.

Browse other questions tagged .