-1
$\begingroup$

i'm trying to solve a problem in which i need to carry out reinforcement learning with multiple simultaneous actions in continuous action space . i checked the multiagent structure; however, im trying to solve a problem in which there are difficulties to set up connection between the agents. for instance, they should take actions simultaneously so there is no way they can be aware of each other's actions. so i decided to go with the multivariate normal solution. has anybody tried that out ever?

first of all i have have difficulties finding the covariance matrix. since it has to be PSD so i decided to assume covariance is zero. something like:

covariance matrix = [[variance1 0][0 variance2]]

but its not everything. the agent doesn't seem to be learning. the problem to be solved by the agent is about resource allocation so the "mean" can not be negative then i decided to go with the "RELU" activation function for the neural network. surprisingly, mean is usually zero so as you can guess its updating the weights in a way to do nothing (negative mean). on the other hand, the variances are on the rise. Though i have checked it a million times there might be a flaw on the code of the environment there is no doubt. i just wanted to to make sure if its mathematically ok to go in this way ? i checked for papers and i found bunch of them but they don't seem to be enough. i would appreciate any guidance.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

Sounds like you have several problems with the way your policy is parametrized.

You don't have to use the multivariate normal distribution. It can work, and probably others have done it already (if not with AAC, surely with DDPG, as it'll be easier to derive the policy gradient there). I won't explain how to use the multivariate normal with either case as there is a much simpler solution for the AAC.

Just have a regular normal distribution over each action. Then, you estimate the mean and variance separately for each action. In other words, it's exactly the same as if you had a single action, but instead of outputting a single mean/variance, you output a vector of mean/variance. Note that this is the same as a multivariate normal distribution, with a diagonal covariance matrix.

Improve this answer
$\endgroup$
3
  • $\begingroup$ thanks for the response. i implemented the way you depicted and it worked so i appreciate your guidance. just to be sure please correct me if i'm wrong. right now i have two policy networks each estimating mean and variance of an action separately. however, i want the actions to be taken, taking into account the fact that their sum should not pass a limit. clearly, i can not handle this limitation through objective function of the policy networks so i'm penalizing the actions with such characteristics through reward. moreover, i make connection between the two policy networks through state. $\endgroup$
    – navid mohamadi
    Commented Jun 7, 2021 at 6:21
  • $\begingroup$ You probably want to have a single network for the actions. So the output layer would be dimension 4, for the mean/variance for both actions. As for your additional constraint on the sum of actions, this is a bit more tricky. Using the penalty seems reasonable. You could also try manually clipping the actions if they are over the sum. $\endgroup$
    – Taw
    Commented Jun 7, 2021 at 14:34
  • $\begingroup$ so would it be fine to modify the objective function of the policy network in this way? "[log(pi(a1))+log(pi(a2))]*A(s,a)" since we are assuming the actions are not correlated we can write it in this way " log(pi(a1,a2))*A". then, because a1 and a2 are not correlated, joint probability of "pi(a1,a2)" can be written as " pi(a1).pi(a2)". then, we can expand the "log(pi(a1).pi(a2))" as "log(pi(a1)) + log(pi(a2))". taking into account that pi(a1) will be calculated with mean1 and varince1 and pi(a2) will be calculated with its related mean and variance. $\endgroup$
    – navid mohamadi
    Commented Jun 8, 2021 at 9:41

You must log in to answer this question.