# How to use a heuristic policy to increase sample efficiency of a deep reinforcement learning agent?

I have a heuristic solution to a problem which works quite well when certain environmental parameters are known and unchanging. However, in a real world setting these parameters will not be known and are likely to fluctuate over the course of an episode. I'm hoping to use deep RL to develop a policy that will be similar to the heuristic, but robust to these unknowns.

My question is: does the RL agent need to be trained "from scratch" as one would typically do or is there a way to leverage the existing policy to jump start the training progress?

In the latter case, what would this looks like? I've had a couple of thoughts, but I'm not sure how well any of them would work.

1. Reward actions that the heuristic would take in an environment with static parameter values, then gradually make the environment more complex and set a new reward function based on what I'm actually interested in.

2. Instead of taking random actions in the exploration stage, take actions dictated by the heuristic.

What you're looking for is called Imitation Learning (IL), in which we are interested in learning an expert policy $$\pi_*$$ which we assume to be optimal.

However, there are many different ways we can approach such learning setting. Just to give some examples, we might be interested in Behavioural Cloning, where our parametrised policy $$\pi_\theta$$ (the RL agent) is trained in a supervised fashion to mimic the expert behaviour on a set of demonstrations $$\mathcal{D} = \{(s_i, \pi_*(s_i))\}^N_{i=1}$$:

$$\theta^* = argmin_\theta \sum_{s \in \mathcal{D}}\mathcal{L}(\pi_*(s), \pi_\theta(s))$$

where $$s_i$$ represents a state and $$\pi(s_i)$$ the action taken by the policy.

Other approaches involve learning a reward function $$\mathcal{R}_\phi$$ from $$\mathcal{D}$$ and use it to reward our agent (Inverse Reinforcement Learning) or GAIL (Generative Adversarial Imitation Learning). I suggest you take a look at this brief overview.

So, to answer your question: yes, you can leverage expert behaviour in your RL algorithm. Should you do so? It depends. IL is not always advisable as the agent might get stuck in a suboptimal policy, especially if the expert you're learning from is poor.

You can also combine IL and RL by starting with the former early on the training and then switch to the latter for later epochs.

Unfortunately, no way of knowing which works best a priori.

Hope this was useful.