3
$\begingroup$

I have a reinforcement learning environment with sparse rewards. Current methods such as PPO and DQN both fail to learn a policy that is suffuciently good. I may have a way to find trajectories that are satisfactory and are found while using no neural network. Is it possible that I then use those trajectories in the replay buffer of a DQN or PPO and then update the neural network a couple of times? This would initialise the neural network in kinda the right direction and would be a bit like imitation learning. Does someone know if this is a viable idea or if there is another way of doing this?

Improve this question
$\endgroup$
1
  • $\begingroup$ PPO no, DQN yes (though there is a caveat that DQN might still not work, particularly if you’re stopping all online interactions with the environment) $\endgroup$
    – David
    Commented Sep 4, 2023 at 14:59

1 Answer 1

Reset to default
2
$\begingroup$

PPO is an on-policy algorithm so you must use trajectories generated by the current policy.

DQN is an off-policy algorithm, so you could add these trajectories to the buffer, but you also need "bad" trajectories so that you try to cover the state-action space. Of course you can't cover the full space, but you need both good and bad examples.

If you have a way of generating "good" trajectories, then maybe try some imitiation learning algorithms like behaviour cloning or DAgger.

You could also try using imitation learning only for pre-training, and once your policy is good enough to generate "reasonable" trajectories, you can train further with PPO.

Improve this answer
$\endgroup$
2
  • $\begingroup$ How would you continue training with PPO afterwards? If you setup a DQN to do the imitation learning part then how do you copy those parameters into the PPO network. I think I am able to generate a good amount of pretty good trajectories for my problem. What would be the difference between using those in a DQN network versus using the imitation learning you have suggested? $\endgroup$
    – Erik Storm
    Commented Sep 5, 2023 at 17:10
  • 1
    $\begingroup$ Imitation learning is different from DQN. In DQN you train a Q-network using the replay buffer. With IL you will train a policy network using supervised learning (see Behavior cloning or DAgger). Once you finish training with IL you can start training with PPO. The PPO algorithm does not require you to start with a random policy. You can start with any policy. In particular you will start PPO with the policy that was trained with IL. $\endgroup$
    – pi-tau
    Commented Sep 5, 2023 at 20:41

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .