I'm working on a reinforcement learning project where I only have demonstrations (i.e. set of states and actions). During my research on how handle the reward signal, I noticed that research papers often design their reward functions, based on heuristics and human/expert knowledge. Meanwhile, when I read papers on Inverse RL, they claim that such approach can be used in applications where designing the reward function is not trivial, although this approach is not common. I'm not experienced in the field, but I'm wondering how one can decide whether to go for designing the reward function or using IRL, instead?
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
It depends on the domain you are in.
Inverse RL (IRL) would be most advantageous in domains in which:
- It's hard to specify the reward by hand: for example, it would be hard to hand-specify a reward for "safe driving", and
- Demonstrations from experts are available: one could find "safe drivers" and collect some of their data, and then use IRL to obtain a more nuanced reward than what we could hand-specify.
That being said, IRL can't provably recover the true reward function (many reward functions will always be consistent with the observed data), and it tends to be very computationally intensive. This is why it's not used very often in practice.
If it's easy to specify the reward (e.g. "win at chess"), hand-specifying it is the simplest way to go about it.
See this related question about ways to hand-design reward functions.
