# How is GARB implemented in PGRD-DL to calculate gradients w.r.t. internal rewards?

In section 3 of this paper the author outlines how GARB was adapted to reduce the variance in updating parameters to an internal reward function estimator.

I have read it a number of times and understand it up through the end of the explanation of GARB. The author then goes on to explain how they use backprop to implement this procedure, which is the point at which I stop understanding.

Is there an open source implementation available to look at? I can’t figure out if $$g_T$$ is actually computed and used or not? And not certain how the internal reward gradient is calculated.

Any insight you can provide would be helpful.

EDIT: after reading it several times and several related papers, I think I have more understanding but not quite there. So my main questions are:

1) are we keeping a full eligibility trace vector with the dimensionality of the vector equal to the number of parameters in $$\theta$$ (all NN params)?

2) do we use the gradient calculation via backdrop at every step to calculate $$g_t$$?

3) do we have to maintain $$3 * \theta$$ parameters, one for theta, one for $$e$$ and one for $$g_t$$?

3) what then is the procedure at terminal $$g_T$$ to update the parameters? A simple element wise matrix operation?

4) How often to update the parameters of $$\theta$$?

• Hi Hanzy! Please, ask one question per post. You're asking too many questions and thus you will make the life of possible answerers harder. Split this post into multiple ones: one post for each of your doubt, unless the questions are very related. – nbro May 6 at 14:43
• @nbro ok will do. My concern is I don’t know how widely read the paper is, so wasn’t sure if it would be helpful or not to break it up. I’ll edit later today. – Hanzy May 6 at 14:48