I am trying to implement the DDPG algorithm.
Why is the actor's loss calculated as the negative mean of the model's predicted Q values in the states we are in? Shouldn't it be like the difference of the Q values when random action is taken and Q values of the model's predicted actions in that state?
Basically, if a random action is generating a better Q value, only then the actor Q value should be updated, else we already have the value and no need to propagate actor loss separately. It will be taken care of in the value loss.
As I understand it, there may be a case when we want to maximize the actor Q values with a constraint given by backpropagation of value loss in the critic network. Is it correct?