2
$\begingroup$

The loss of policy gradient:

$$ \nabla_{\theta}J(\theta)=\mathbb{E}_{\tau\sim\pi\theta}\left[\sum_{t=0}^{T}(R_t-b_t)\nabla_{\theta}\log\pi_\theta(a_t|s_t)\right] $$

where $\pi_\theta(a_t|s_t)$ is the probability of the model $\theta$ predicting the action $a_t$ in the state $s_t$.

Howerver, if the model always predicts a certain incorrect action with a probability of 1, then it necessarily follows that $\log\pi_\theta(a_t|s_t)=0$. So the loss is alwasy 0.

I discovered this issue because I found that my model always overfits by outputting any action with a probability of 1, and there's a high chance it's an incorrect action. At this point, the loss is 0, and convergence stops. What could be the problem?

$\endgroup$

1 Answer 1

2
$\begingroup$

"So the loss is always 0" doesn't mean that there is no learning... indeed, it's called "policy gradient", not "policy loss", thus you have to consider the gradient of the logarithm evaluated at 1 (since $\pi(a|s) = 1)$)

And indeed, $\nabla \ln \pi(a|s)$ when $\pi(a|s) = 1$ is $1$, thus learning can definitely still happen (though it might be slow)

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .