1
$\begingroup$

I'm using Python and tensorflow to implement a Deep Q-learning with experience replay in a continous action and state spaces and I have used two neural networks to approximate both the policy function and the Q-function. While for the Q-function I have a target so I can update the model minimizing the loss function, the policy update pseudocode does not include a target which I can use to fit the model but only a gradient ascent step, as the following

enter image description here

How can I use tensorflow and a sequential model in python to make this update?

Improve this question
$\endgroup$
4
  • $\begingroup$ If you have a policy function, you aren't doing Q-learning any more, but some form of Actor-Critic. Could you please link the source material that you are attempting to follow? The equation you have given looks like a definition of the gradient as opposed to a an explanation on how to sample it and get a meaningful value. Also, as you have a continuous action space, if you want a practical example you will need to explain what your policy function outputs (be prepared to be told that your polciy function is not usable if for instance it is deterministic but you are atttempting A2C). $\endgroup$ – Neil Slater May 23 '20 at 10:12
  • $\begingroup$ Programming issues are off-topic. Currently, the formulation of your question seems to indicate that you want us to provide code. If that's not the case, please, edit your post to clarify that this is a theoretical question. Also, please, read our on-topic page: ai.stackexchange.com/help/on-topic. $\endgroup$ – nbro♦ May 23 '20 at 10:46
  • $\begingroup$ Thanks I'll ask a new question $\endgroup$ – aandre_90 May 23 '20 at 11:05
  • $\begingroup$ @aandre_90 Please, tag me with @nbro, so that I receive a notification of your message/comment. $\endgroup$ – nbro♦ May 23 '20 at 11:18