There are several different angles we can classify Reinforcement Learning methods from. We can distinguish three main aspects :

  • Value-based and policy-based
  • On-policy and off-policy
  • Model-free and model-based

Historically, due to their sample-efficiency, the model-based methods have been used in the robotics field and other industrial controls. That is happened due to the cost of the hardware and the physical limitations of samples that could be obtained from a real robot. Robots with a large number of degrees of freedom are not widely accessible, so RL researchers are more focused on computer games and other environments where samples are relatively cheap. However, the ideas from robotics are infiltrating, so, who knows, maybe the model-based methods will enter the focus quite soon.

As we know, "model" means the model of the environment, which could have various forms, for example, providing us with a new state and reward from the current state and action. From what I have seen so far, all the methods (i.e. A3C, DQN, DDPG) put zero effort into predicting, understanding, or simulating the environment. What we are interested in is proper behavior (in terms of the final reward), specified directly (a policy) or indirectly (a value), given the observation. The source of observations and reward is the environment itself, which in some cases could be very slow and inefficient.

In a model-based approach, we're trying to learn the model of the environment to reduce the "real environment" dependency. If we have an accurate environment model, our agent can produce any number of trajectories that it needs, simply by using the model instead of executing the actions in the real world.


I am interested in a day trading environment. Is it possible to create a model-based environment in order to build an accurate day trading environment model?

  • $\begingroup$ Probably not, but if you believe there is sufficient data only within the stock prices you can train off prior data under the assumption your trades won't impact the market $\endgroup$ May 27, 2020 at 2:10
  • $\begingroup$ @FourierFlux This is a bit short to make a good answer. Perhaps you could expand on why modelling day trading is hard. The OP does not suggest that the model needs to be intrinsic to price variations only (in fact they don't suggest anything at all about the model). $\endgroup$ May 27, 2020 at 7:35
  • $\begingroup$ You give some decent generic background which seems accurate to me, and I think it is useful. However, your question then asks something specific without much clue on where you are with your knowledge or project. I think it would help if you expand giving some details about how you are addressing the question - what do you know about modelling your target environment of day trading and what have you tried? $\endgroup$ May 27, 2020 at 7:39


You must log in to answer this question.