0
$\begingroup$

I'm a newbie to reinforcement learning. While studying reinforcement learning, a question arose about how to apply reinforcement learning in the real world.

Assuming that a reinforcement learning agent trained in simulation is used in the real world, do they usually only use the optimal policy learned in the simulator? Or are they need additional exploration (learning) in the real world?

In my view, the latter approach is quite impractical in real-world environments where failure by agents' actions is fatal. However, it also seems very difficult for the simulation environment to reflect the real world.

What approach is being adopted by the industry field?

Improve this question
$\endgroup$
1
  • $\begingroup$ Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. $\endgroup$
    – Community Bot
    Sep 8, 2022 at 9:07

1 Answer 1

Reset to default
1
$\begingroup$

RL is not used much in the real industry, as you said because of safety concerns. There are perhaps three different ways of how this would be possible

  1. Use safe exploration to learn a model of the environment and apply model predictive control algorithms on the learned model. This seems to be the approach taken for data center cooling using model-predictive control.

  2. Design an accurate simulation model as a digital twin and use it to train policies there before deploying into the real world. This was the approach taken in the recent breakthrough in nuclear fusion: Magnetic control of tokamak plasmas through deep reinforcement learning

  3. Use existing controllers to generate data sets and train RL agents with offline RL (or perhaps imitation learning) and use it as a prior before finetuning it in the real world.

Now, as you mention there is the sim-to-real problem if you train in simulation. The real world usually comes with various uncertainties that can not be simulated well. This may be not extremely relevant for controlled industrial environments but is important in robotics. We can divide the training process into multiple phases, where the agent has less and less information available. The idea is to train a belief model, which aims to assess all the uncertainties and safely adapt to them. An example is: Learning robust perceptive locomotion for quadrupedal robots in the wild

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .