Using states (features) and actions from a heuristic model to estimate the value function of a reinforcement learning agent [closed]

Question

new to RL here.
As far as i understood from RL courses, that there is two sides of reinforcement learning. Policy Evaluation, which is the task of knowing the value function for certain policy. and Control, which is maximizing the reward or the value function. what if i have a heuristic agent that performs almost acceptable performance in an environment but i want to find a policy that tends to be the optimal policy, is there a way to cut the first half of the task by teaching the agent ? will be a side by side buffer of the (states, actions) be sufficient ?

Answer 1

Not sure I fully understand your question, are you asking:

1. if you can skip the policy evaluation part?
2. if you can speedup the training of your agent by mimicking the behaviour of your heuristic agent first and then optimise from there?

If the first, I am not sure exactly what you are asking as most RL algorithm doesn't have an explicit policy evaluation phase and even for ones which does (policy iteration for example) you would be using the wrong policy to evaluate.

If you are asking about the second one (warm start your agent) that's very much possible. You would do it differently based on the algorithm:

• With DQN, you have your buffer which contains (state, action, next_state, reward) tuples. You could just let loose your heuristic agent in your environment and record these tuples and put them into the buffer. Then you could train your DQN agent offline (without interacting with the environment) for a while until it performs roughly as well as your heuristic one and then you could let it loose in your environment so it can get better.
• With something like an Actor Critic you might not have a buffer so the process is slightly different. You also have two components, the policy function and the value function. You can train the policy function to mimic what your heuristic agent does by gathering lot of experience from the heuristic agent and basically do supervised learning on that experience. You would also need to train your value function but that would be very similar to it's normal training.

Hope this helps, let me know if anything is unclear.