0
$\begingroup$

I am a beginner in the field of Reinforcement Learning with only a couple of months of experience being in the field. Soon, I will start working on a project where we want to optimize the production of a chemical unit through reinforcement learning approach. From the SME's, we already obtained a simulator code that can take some input and render us the output. A part of our output is our objective function that we want to maximize by tuning the input variables. From a reinforcement learning angle, the inputs will be the agent actions, while the state and reward can be obtained from the output. We are currently in the process of building a RL environment, the major part of which is the simulator code described above.

We were talking to a RL expert and she mentioned that one of the thing that we have here conceptually wrong is that our environment will not have the Markov property in the sense that it is really a 'one-step process' with the process not continuing from the previous state and there is no sort of continuity in state transitions. She is correct there. This made me think, how can we get around this then. Can we perhaps append some part of the current state to the next state etc. More importantly, I have seen RL applied to optimal control in other examples as well which are non-markovian ex. scheduling, tsp problems, process optimization etc. What is the explanation in such cases? Does one simply assumes process to be markovian with unknown transition function?

Pardon me if my question appears too open ended but I am keen to seek some advice through this forum and would like to receive pointers helpful to think in right direction. Also, I think many others will find this question relevant too in the future.

$\endgroup$
1
  • 1
    $\begingroup$ The Markov Property is an extremely hard constraint that is often not met in real life application. Many RL methods will work fine without the Markov property. A simple way is to add previous states in the value and action fonctions for instance. Given that, I think that you should either give more details on your specific problem or rephrase the theoretical part you are not understanding. $\endgroup$ – Romain Mar 26 at 18:57
1
$\begingroup$

RL is currently being applied to environments which are definitely not markovian, maybe they are weakly markovian with decreasing dependency.

You need to provide details of your problem, if it is 1 step then any optimization system can be used.

$\endgroup$
1
  • $\begingroup$ what do you mean by 1 step? Currently yes, the simulator can take a set of input and render output. But there is nothing dynamic and time-dependent in it. Like I mentioned the next state (output) then is only being determined here by the next action (input) which I am free to choose and as such has got nothing to do with the previous state. $\endgroup$ – your_boy_gorja Mar 26 at 5:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.