There are several different angles we can classify Reinforcement Learning methods from. We can distinguish three main aspects :
- Value-based and policy-based
- On-policy and off-policy
- Model-free and model-based
Historically, due to their sample-efficiency, the model-based methods have been used in the robotics field and other industrial controls. That is happened due to the cost of the hardware and the physical limitations of samples that could be obtained from a real robot. Robots with a large number of degrees of freedom are not widely accessible, so RL researchers are more focused on computer games and other environments where samples are relatively cheap. However, the ideas from robotics are infiltrating, so, who knows, maybe the model-based methods will enter the focus quite soon.
As we know, "model" means the model of the environment, which could have various forms, for example, providing us with a new state and reward from the current state and action. From what I have seen so far, all the methods (i.e. A3C, DQN, DDPG) put zero effort into predicting, understanding, or simulating the environment. What we are interested in is proper behavior (in terms of the final reward), specified directly (a policy) or indirectly (a value), given the observation. The source of observations and reward is the environment itself, which in some cases could be very slow and inefficient.
In a model-based approach, we're trying to learn the model of the environment to reduce the "real environment" dependency. If we have an accurate environment model, our agent can produce any number of trajectories that it needs, simply by using the model instead of executing the actions in the real world.
Question:
I am interested in a day trading environment. Is it possible to create a model-based environment in order to build an accurate day trading environment model?
