From what I have seen, any results involving RL almost always take a massive number of simulations to reach a remotely good policy.
Will any form of RL be viable for real-time systems?
$\begingroup$
$\endgroup$
1
-
$\begingroup$ So do normal ML algos. They take a lot of time to train. RL also takes a lot of time to train. But at testing time you are only doing a single forward propagation which takes very less time compared to the backprop operation in training plus it is performed repeatedly over the data-set. $\endgroup$– user9947Commented Feb 29, 2020 at 14:00
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
7
Short answer: Yes, it is.
Explanation
Reinforcement learning can be considered as a online learning. That is, you can train your model with a single data/reward pairs. As with any online learning algorithm, there are a few things to consider.
The model tends to forget the knowledge gained. To overcome this problem, one can save new data in a circular buffer called history and train the model with a portion of mix of new and old data. This is actually the common way to train an RL model and can be adopted to real-time systems. There are also others techniques to overcome it.
Another problem is that if only one data point is fed to the network, it will be impossible to apply some techniques, such as Batch normalization.
-
1$\begingroup$ This doesn't address the massive, massive number of simulations required. IIRC the number of walking simulations for robots is way, way higher than the real time constraints imposed on animals learning to walk. $\endgroup$ Commented Feb 29, 2020 at 19:43
-
$\begingroup$ Where does this requirement come from? You can pre-train your model and then apply online learning so that it is always up to date. $\endgroup$ Commented Feb 29, 2020 at 19:54
-
1$\begingroup$ The issue is IMO, real systems have built in priors I think(or models which better lead to correct parameter estimation) and it seems this is missing from the RL paradigm. Like the convergence rate for animal/human behavior is astronomically faster than any type of RL policy. $\endgroup$ Commented Feb 29, 2020 at 20:22
-
1$\begingroup$ But do you dispute the fact that humans don't need nearly the same number of trials to reach a viable policy? $\endgroup$ Commented Feb 29, 2020 at 20:45
-
1$\begingroup$ Humans may not need as much online training as we are capable of transferring knowledge from different domains to help in achieving the task $\endgroup$– KaneMCommented Mar 1, 2020 at 12:37