Consider maximizing the function $R(w)$ with parameter $w$ using gradient ascent. However, we don't know the gradient $\nabla_wR(w)$ formula. Now suppose $w$ is sampled from a probability distribution $\pi(w,\theta)$ parameterized by $\theta$. Then we can define

$$J(\theta)=E[R(w)]=\int R(w)\pi(w,\theta)dw.$$ And we have

$$\nabla_\theta J(\theta)=E[R(w)\nabla_\theta \log \pi(w,\theta)]$$.

Then, if we sample $w_1,\ldots,w_N$, we can estimate the gradient as $$\nabla_\theta J(\theta)\approx \frac{1}{N}\sum_{i=1}^N R(w_i) \nabla_\theta \log \pi(w_i, \theta).$$

It looks like REINFORCE algorithm in Deep Reinforcement Learning. Does this algorithm have a name? Is the above derivation correct?

I wonder if it is useful in optimizing $R(w)$ function.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.