7
$\begingroup$

I am studying the state of the art of Reinforcement Learning, and my point is that we see so many applications in the real world using Supervised and Unsupervised learning algorithms in production, but I don't see the same thing with Reinforcement Learning algorithms.

What are the biggest barriers to get RL in production?

$\endgroup$
0
7
$\begingroup$

There is a relatively recent paper that tackles this issue: Challenges of real-world reinforcement learning (2019) by Gabriel Dulac-Arnold et al., which presents all the challenges that need to be addressed to productionize RL to real world problems, the current approaches/solutions to solve the challenges, and metrics to evaluate them. I will only list them (based on the notes I had taken a few weeks ago). You should read the paper for more details. In any case, for people that are familiar with RL, they will be quite obvious.

  1. Batch off-line and off-policy training
    • One current solution is importance sampling
  2. Learning on the real system from limited samples (sample inefficiency)
    • Solutions: MAML, use expert demonstrations to bootstrap the agent, model-based approaches
  3. High dimensional continuous state and action spaces
    • Solutions: AE-DQN, DRRN
  4. Satisfying safety constraints
    • Solutions: constrained MDP, safe exploration strategies, etc.
  5. Partial observability and non-stationarity
    • Solutions to partial observability: incorporate history in the observation, recurrent neural networks, etc.
    • Solutions to non-stationarity: domain randomization or system identification
  6. Unspecified and multi-objective reward functions
    • Solutions: CVaR, Distributional DQN
  7. Explainability
  8. Real-time inference
  9. System delays (see also this and this answers)

There's also a more recent and related paper An empirical investigation of the challenges of real-world reinforcement learning (2020) by Gabriel Dulac-Arnold et al, and here you have the associated code with the experiments.

However, note that RL (in particular, bandits) is already being used to solve at least one real-world problem [1, 2]. See also this answer.

$\endgroup$
4
$\begingroup$

Technical barriers: There should be at least these common sense big barriers:

  • Trial-and-error technique makes the model hard to learn (too many), compared to ready-to-use supervised data
  • Number of time-steps (which usually equals the number of actions of the agent in the trajectory) is large, thus brute-force exploration won't work as the number of trials to find errors is exponential, although negative rewards may help cut short the brute-force tree.
  • Real-life RL takes unlimited number of episodes (for each episode, a sequence of actions should be learnt), and the incremental training is harder and harder in time with more explored data, unless some past and no-longer-related data are removed, just like humans, we forget some of the past to learn more, remember more the present.

The technical barriers are at first the barriers to applying them to business. People may produce some supervised data manually rather quick, and thus supervised learning is usually opted first, nobody wish to try RL.

Harder to find human resources: AI engineers with experiences in supervised learning are more popular and easier to find some; fewer work with RL, thus business projects are not carried out easily if using RL.

However, from my point of view, RL is very much promising in future as AI entities are now more and more on their own.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.