2
$\begingroup$

At the beginning, if I have a complete model $p(s' \mid s, a)$ (an assumed true model that describes the environment well enough) and the reward function $r(s,a,s')$. How can I exploit the model and learn a good policy in this situation? Assume that the state space is continuous, and the action space is finite.

Traditional dynamic programming/planning methods like policy iteration or value iteration cannot be directly applied since there are infinitely many states. If I try to get samples from the model and apply an algorithm like DQN or any non-linear function approximation, it looks like that I use the model-free approach and do not get the full advantages of the model.

May I ask if there are any reinforcement learning/planning methods that use the explicit model given at the beginning to solve the MDP? (in case of continuous state space and finite action space)

$\endgroup$
4
  • $\begingroup$ Are you asking which model-based RL algorithms exist (assuming that you don't consider value/policy iteration an RL algorithm but a planning algorithm)? $\endgroup$
    – nbro
    Apr 17 at 12:11
  • $\begingroup$ @nbro Yes, kind of. I understand that policy/value iteration are planning algorithms. But in Sutton's book it's only for finite state space. Assuming that the state space are continuous, and the transition function $p$ and reward function $r$ are explicitly given to the agent at the beginning, is there any method/algorithm to use the model to find a good or optimal policy? $\endgroup$
    – k2pctdn
    Apr 17 at 12:23
  • $\begingroup$ Please, edit your post to include this info and change the title to be more specific and have your question $\endgroup$
    – nbro
    Apr 17 at 12:24
  • $\begingroup$ @nbro thank you for your editing, I've also added some more information to the post. $\endgroup$
    – k2pctdn
    Apr 17 at 15:11

1 Answer 1

2
$\begingroup$

In optimal control field to minimize certain well-defined costs especially in process industries, continuous state space model-based planning methods such as model predictive control (MPC) is a common decision time planning/control method which can handle both linear and nonlinear models, and unlike training data intensive model-free RL methods it's much more data efficient, faster, online, and good (though may not optimal):

MPC is based on iterative, finite-horizon optimization of a plant model. At time $t$ the current plant state is sampled and a cost minimizing control strategy is computed (via a numerical minimization algorithm) for a relatively short time horizon in the future:$[t,t+T]$. Specifically, an online or on-the-fly calculation is used to explore state trajectories that emanate from the current state and find (via the solution of Euler–Lagrange equations) a cost-minimizing control strategy until time $t+T$. Although this approach is not optimal, in practice it has given very good results. Much academic research has been done to find fast methods of solution of Euler–Lagrange type equations, to understand the global stability properties of MPC's local optimization, and in general to improve the MPC method.

If your model is linear (most mechanical systems) and cost is quadratic, LQR is also a common optimal feedback control method.

The main differences between MPC and LQR are that LQR optimizes across the entire time window (horizon) whereas MPC optimizes in a receding time window,[4] and that with MPC a new solution is computed often whereas LQR uses the same single (optimal) solution for the whole time horizon.

Finally similar to Dyna-Q learning architecture with background learning & planning introduced in Sutton and Barto's Reinforcement Learning, An Introduction, especially if your initial known sample model is not accurate or keeps evolving, one can combine model-based with model-free RL methods, you just replace tabular Q-planning there with your linear or nonlinear function approximations planning version such as described in Jong and Stone's paper Model-Based Exploration in Continuous State Spaces:

This paper develops a method for approximating continuous models by fitting data to a finite sample of states, leading to finite representations compatible with existing model-based exploration mechanisms. Experiments with the resulting family of fitted-model reinforcement learning algorithms reveals the critical importance of how the continuous model is generalized from finite data. This paper demonstrates instantiations of fitted-model algorithms that lead to faster learning on benchmark problems than contemporary model-free RL algorithms that only apply generalization in estimating action values. Finally, the paper concludes that in continuous problems, the exploration-exploitation tradeoff is better construed as a balance between exploration and generalization.

$\endgroup$
1
  • $\begingroup$ Thank you, your answer clarify a lot. $\endgroup$
    – k2pctdn
    Apr 18 at 7:15

You must log in to answer this question.