I was reading up a paper that did routing based on an MDP, and I was wondering because, in routing, there is a sender node and a receiver node, so if the receiver node changes (sending a message to someone else), would we have to train the MDP algorithm all over again?

This also got me thinking about what would happen even if one node in the process of transmission changes. Does using an MDP for training the agent mean that the obstacle and goals should never change?

  • $\begingroup$ Technically you do not "train" an MDP, unless you are asking about inverse reinforcement learning. I do not think you are, but not 100% sure, so please clarify. Usually you train an agent which learns to optimise reward when making action choices within an MDP. In which case your question is about whether to retrain the agent as opposed to the MDP $\endgroup$ – Neil Slater May 3 at 8:29
  • $\begingroup$ Made the relevant changes, thanks for pointing it out $\endgroup$ – Ravish Jha May 3 at 8:31
  • $\begingroup$ To have more context, could you please provide the name of and link to the paper you were reading? Moreover, you may want to reformulate your question because saying "an agent trained using MDP" or "using an MDP for training" does not make much sense, i.e., even after your last changes, the post is still not "ok", in my view. The MDP is just the mathematical model of the environment. The solution to the MDP is a policy. RL algorithms are used to find a policy. So, based on this info, you may want to formulate your post again to make it more meaningful. $\endgroup$ – nbro May 3 at 12:17

It is possible, at design time for a reinforcement learning problem, to allow for changes within an environment. You can make any element into a variable property of the state, that the agent can realistically be told at the start or sense from the environment.

If you do add new variable to model the possibility of change:

  • It allows the agent to learn to solve a more general problem where the chosen property can vary.

  • It increases the size of the state space.

  • It requires training to include variations of the new variable.

Usually this also increases the time taken to train.

It is not always possible to use a state variable for the task - perhaps a goal state is effectively hidden from the agent and the purpose of training is for it to be discovered. In which case, you will require at least some re-training. It may be faster to start with the existing trained agent if the difference is not large.

If you cannot simply extend the state representation, and the environment changes in a small enough way, then it may also be possible to use an agent which continuously explores which will re-train itself over time in repsonse to changes in the environment. The DynaQ+ algorithm is an example of a method which is designed to explore and find changes in the agent's environment to allow for this kind of online retraining when things change.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.