In section 3 of this paper the author outlines how GARB was adapted to reduce the variance in updating parameters to an internal reward function estimator.
I have read it a number of times and understand it up through the end of the explanation of GARB. The author then goes on to explain how they use backprop to implement this procedure, which is the point at which I stop understanding.
Is there an open source implementation available to look at? I can’t figure out if $g_T$ is actually computed and used or not? And not certain how the internal reward gradient is calculated.
Any insight you can provide would be helpful.
EDIT: after reading it several times and several related papers, I think I have more understanding but not quite there. So my main questions are:
1) are we keeping a full eligibility trace vector with the dimensionality of the vector equal to the number of parameters in $\theta$ (all NN params)?
2) do we use the gradient calculation via backdrop at every step to calculate $g_t$?
3) do we have to maintain $3 * \theta$ parameters, one for theta, one for $e$ and one for $g_t$?
3) what then is the procedure at terminal $g_T$ to update the parameters? A simple element wise matrix operation?
4) How often to update the parameters of $\theta$?