I'm trying to come up with a loss function for the case, in DDPG, where we have as many outputs from the critic as there are from the actor. So, there will be one Q value for each dimension in the actor's output.

Currently, I would have a separate loss function for each of the output dimensions, but this doesn't scale. Also, to shed more light, I have as output from the actor two output layers. One contains the actions, but rather than be continuous I have a SoftMax function on it. While the other contains a conditioning vector that I will use to condition the observations of the next time step. The conditioning vector also has a SoftMax, and I convert the top 2 % values to ones while keeping the rest as zeros before conditioning. The reason I use DDPG rather than DQN for this, even though it's a discrete action space, is so that I can have the binary vector besides the actions as output.

So, one loss function for this ensemble would go a long way to helping me out. So, this time there will be as many outputs from the critic as there are dimensions in the binary vector, +1 for the action segment.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.