I've implemented a vanilla actor-critic and have run into a wall. My model does not seem to be learning the optimal policy. The red graph below shows its performance in cartpole, where the algorithm occasionally does better than random but for the most part lasts between 10-30 timesteps.

enter image description here

I am reasonably sure that the critic part of the algorithm is working. Below is a graph of the delta value (r + Q_w(s',a') - Q_w(s,a)), which seems to show that for the most part the predicted future reward is quite similar to the approximate future reward.

enter image description here

Thus, I am at a loss for what the problem could be. I have an inkling it lies within the actor, but I am not sure. I have double checked the loss function and that seems to be correct to me as well. I would appreciate any advice. Thanks!


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.