1
$\begingroup$

I have PPO agent for discrete action space for LunarLander-v2 env in gym and it works well. However, when i am trying to solve continuous version of the same env - LunarLanderContinuous-v2 it is totally failing. I guess i made some mistakes in converting algorithm to continuous version. So, my steps of changing at algorithm are:

  1. Change network: return mu and var of shape n_actions. I have 2 different last layers for that, for mu i return Tanh of logits and for var i return Softplus of logits.
  2. Choosing action: sampling action from normal distribution with expectation mu and variance var - torch.distributions.multivariate_normal.MultivariateNormal(torch.squeeze(mu), torch.torch.diag_embed(var))
  3. For log of action probability i am using dist.log_prob(actions)

With this small changes my algorithm totally doesn't work. Is it right steps to convert algorithm with discrete action space to algorithm with continuous action space? I really confused, because my PPO for discrete action space work very well and with only this changes it is failing. Could you please suggest what i am doing wrong here?

Improve this question
$\endgroup$
5
  • $\begingroup$ It sounds like you did the right changes though I will look more closely at changes I did with the same environment and algorithm. One thing that might be happening is that learning with continuous actions is just harder and you could have the wrong parameters. I think you should try changing the learning rate and other hyper parameters, increase the learning time, and maybe keeping the variance constant to make sure it’s not a problem with your code. $\endgroup$ – S2673 Apr 28 at 15:16
  • $\begingroup$ @S2673 Thank you for response! I found a solution but unfortunately i haven't found reason yet. So, if i have 2 distinct neural networks for 2 actions it works really fine. But, if i have at least 1 common layer(for example first layer is the common one, and 2 next layers are different for 2 action, so i have like 5 layers in my network and 2 tails, sorry if i confused:)) all just stop working at all! It is really curios for me why it happens and may be there are some papers that checking for this problem? About problem - continuous lunar lander. $\endgroup$ – Alexander Yukhimchuk Apr 30 at 6:42
  • $\begingroup$ I don’t think I understand exactly, but it sounds like the problem is with the code (especially in the neural network) instead of the parameters. It still could be possible that the parameters work only for simpler networks with one action output. Try changing the learning rate and size and amount of hidden layers and make sure the code is correct. I think I have PPO code somewhere that works with LunarLander and I could share it with you if you want. $\endgroup$ – S2673 May 1 at 18:59
  • $\begingroup$ @S2673 Yes, please can you share your network, thanks! $\endgroup$ – Alexander Yukhimchuk May 3 at 12:36
  • $\begingroup$ I put the code in here. It is written in Python with PyTorch and there are no explaining comments. You probably want to focus on the hyperparameters, and the Agent class, especially the initialization, storage, and updating. Run exactly as it is, it did pretty well on continuous LunarLander after 50 training loops. Holding space at any time will stop the training. agent.run(1) will animate the environment and show you how the agent is doing and agent.train(50) will train for 50 training loops. $\endgroup$ – S2673 May 3 at 23:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.