I am new to this group, Anybody familiar with Q-learning algorithm and RMSprop approach ? i have a question regarding the application of RMSprop approach into Q-Learning to adapt dynamically the learning rate for each Q(s,a). I am confused how to compute the average square gradient in tabular Q-Learning. Is it E[g^2] = beta * E[g^2]+ (1-beta) * td_error**2. ? and what would be the formula to update Q(s,a) ? Is it Q(s,a) <- Q(s,a) + alpha_0 * td_error/np.sqrt(E[g^2]+epsilon) ? Thank you for help in advance ?
1
-
$\begingroup$ Can you please edit your post to use mathjax to format the math formulas? $\endgroup$– nbroCommented Dec 11 at 11:25
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
1
If you want to use RMSProp for a tabular Q-learning algorithm, then you will have to first identify what the gradient is (the update rule - i.e. the gradient ascent - for Q-learning clearly suggests that the td_error is the gradient). Then, you just apply that to RMSProp by replacing the gradient g by the td_error in all RMSProp equations. So, what you are suggesting seems reasonable to me!
The only thing I can find to say is that you should be careful to keep a different moving average E[g^2] for each state-action pair (i.e. E[g^2] becomes a table, of the same size as Q)! (Note that the td_error is also state-action-dependent!) So, I would guess something like (with E[g^2] as M, for moving average):
M(s, a) <- beta * M(s, a) + (1-beta) * td_error(s, a)**2
Q(s,a) <- Q(s,a) + alpha_0 * td_error / np.sqrt(M(s, a)+epsilon).
-
$\begingroup$ Thanks for your reply. I applied this two equations in my q-update function. However, it didn't work for me. Starting alpha at 0.001, i was expecting its value to increase over time, but still always at 0.0001 with very small variation $\endgroup$ Commented Nov 11 at 12:58