2
$\begingroup$

Following Deep Q-learning from Demonstrations, I'd like to avoid potentially unsafe behavior during early learning by making use of supervised learning with demonstration data. However, the implementation I'm following still uses an environment. Can I train my agent without an environment at all?

$\endgroup$
  • $\begingroup$ Most of this question is off-topic for this community, as we don't address implementation details. We can help you understand that paper and general things about environments, though. $\endgroup$ – Philip Raeisghasem Mar 11 at 6:44
  • $\begingroup$ Sorry, I'm newbie here. I addressed the implementation to make people see that people still uses Environments to "learn from Demonstrations" while the devil is in learning only from a certain dataset, isn't it? $\endgroup$ – Angelo Mar 11 at 8:56
  • $\begingroup$ I didn't mention you. Sorry again @PhilipRaeisghasem. $\endgroup$ – Angelo Mar 11 at 9:17
  • $\begingroup$ Hah, it's ok. I'm considering how to best answer your question. $\endgroup$ – Philip Raeisghasem Mar 11 at 9:20
2
$\begingroup$

There are many techniques for training an RL agent without explicitly interacting with an environment, some of which are cited in the paper you linked. Heck, even using experience replay like in the foundational DQN paper is a way of doing this. However, while many models utilize some sort of pre-training for the sake of safety or speed, there are a couple of reasons why an environment is also used whenever possible.

Eventually, your RL agent will be placed in an environment to take its own actions. This is why we train RL agents. I'm assuming that, per your question, learning does not happen during this phase.

Maybe your agent encounters a novel situation
Hopefully, the experience your agent learns from is extensive enough to include every possible state-action pair $(s,a)$ that your agent will ever encounter. If it isn't, your agent won't have learned about these situations, and it will always perform suboptimally in them. This lack of coverage over the state-action space could be caused by stochasticity or nonstationarity in the environment.

Maybe the teacher isn't perfect
If you don't allow your agent to learn from its own experience, it will only ever perform as well as the agent that collected the demonstration data. That's an upper bound on performance that we have no reason to set for ourselves.

$\endgroup$
  • $\begingroup$ Thanks a lot Philip. More specifically, I'm planning to pre-train a recommender system from a generated dataset using user embeddings and last interactions as the state and the items to recommend as the actions to take. Therefore, I'm very concerned about including every possible state-action par, because although the states may be very close one from another, it won't be the same exactly. Does it really matter? $\endgroup$ – Angelo Mar 11 at 10:15
  • $\begingroup$ It depends on how well your model can generalize between similar states. More training data always helps with this. Regularization could also help. But, again, it's very possible your system needs to learn from its own experience--through trial and error after deployment--in order to learn optimal behavior. $\endgroup$ – Philip Raeisghasem Mar 11 at 11:08
  • $\begingroup$ Of course, I'm only talking about the pre-deployment phase. Thanks a lot. $\endgroup$ – Angelo Mar 11 at 13:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.