I'm trying to come up with a loss function for the case, in DDPG, where we have as many outputs from the critic as there are from the actor. So, there will be one Q value for each dimension in the actor's output.
Currently, I would have a separate loss function for each of the output dimensions, but this doesn't scale. Also, to shed more light, I have as output from the actor two output layers. One contains the actions, but rather than be continuous I have a SoftMax function on it. While the other contains a conditioning vector that I will use to condition the observations of the next time step. The conditioning vector also has a SoftMax, and I convert the top 2 % values to ones while keeping the rest as zeros before conditioning. The reason I use DDPG rather than DQN for this, even though it's a discrete action space, is so that I can have the binary vector besides the actions as output.
So, one loss function for this ensemble would go a long way to helping me out. So, this time there will be as many outputs from the critic as there are dimensions in the binary vector, +1 for the action segment.