# How is REINFORCE used instead of Backpropagation?

In neural networks with stochastic layers I've seen the use of the REINFORCE estimator for estimating the gradient (because it can't be computed directly).

Some such examples are Show, Attend and Tell, Recurrent models of visual attention and Multiple Object Recognition with Visual Attention.

However, I haven't figured out how this exactly works. How do we "bypass" the gradient's computation by using the REINFORCE learning rule? Does anyone have any insight on this?

$${\huge \Delta \mathbf{\theta}_t = \alpha \nabla_{\mathbf{\theta}} \log \pi_{\mathbf{\theta}} (a_t \mid s_t) v_t }%$$
As this shows, the gradient is still there ($$\nabla_\theta$$). But the policy corresponds to the network's output, so we can use backpropagation to compute the gradient of that heuristic loss with respect to the weights. The real gradient is unknown to us, but this estimation will do the job.