In optimal control field to minimize certain well-defined costs especially in process industries, continuous state space model-based planning methods such as model predictive control (MPC) is a common decision time planning/control method which can handle both linear and nonlinear models, and unlike training data intensive model-free RL methods it's much more data efficient, faster, online, and good (though may not optimal):
MPC is based on iterative, finite-horizon optimization of a plant model. At time
$t$ the current plant state is sampled and a cost minimizing control strategy is computed (via a numerical minimization algorithm) for a relatively short time horizon in the future:$[t,t+T]$. Specifically, an online or on-the-fly calculation is used to explore state trajectories that emanate from the current state and find (via the solution of Euler–Lagrange equations) a cost-minimizing control strategy until time $t+T$. Although this approach is not optimal, in practice it has given very good results. Much academic research has been done to find fast methods of solution of Euler–Lagrange type equations, to understand the global stability properties of MPC's local optimization, and in general to improve the MPC method.
If your model is linear (most mechanical systems) and cost is quadratic, LQR is also a common optimal feedback control method.
The main differences between MPC and LQR are that LQR optimizes across the entire time window (horizon) whereas MPC optimizes in a receding time window,[4] and that with MPC a new solution is computed often whereas LQR uses the same single (optimal) solution for the whole time horizon.
Finally similar to Dyna-Q learning architecture with background learning & planning introduced in Sutton and Barto's Reinforcement Learning, An Introduction, especially if your initial known sample model is not accurate or keeps evolving, one can combine model-based with model-free RL methods, you just replace tabular Q-planning there with your linear or nonlinear function approximations planning version such as described in Jong and Stone's paper Model-Based Exploration in Continuous State Spaces:
This paper develops a method
for approximating continuous models by fitting data to a finite sample of states, leading to finite representations compatible with existing
model-based exploration mechanisms. Experiments with the resulting
family of fitted-model reinforcement learning algorithms reveals the critical importance of how the continuous model is generalized from finite
data. This paper demonstrates instantiations of fitted-model algorithms
that lead to faster learning on benchmark problems than contemporary
model-free RL algorithms that only apply generalization in estimating
action values. Finally, the paper concludes that in continuous problems,
the exploration-exploitation tradeoff is better construed as a balance between exploration and generalization.
