I just wanted to confirm that my understanding of the different Markov Decision Processes are correct, because they are the fundamentals of reinforcement learning. Also, I read a few literature sources, and some are not consistent with each other. What makes most sense to me is listed below:

Markov Decision Process

All the states of the environment is known so the agent has all the information required to make the optimal decision in every state. We can basically assume that the current state has all information about the current state and all the previous states (i.e., the Markov property)

Semi Markov Decision Process

The agent has enough information to make decisions based on the current state. However, the actions of the agent may take a long time to complete, and may not be completed in the next time step. Therefore, the feedback and learning portion should wait until the the action is completed before being evaluated. Because the action takes many time steps, "mini rewards" obtained from those time steps should also be summed up.

Example: Boiling water.

State 1: Water is at 23 C

Action 1: Agent sets stove top at 200 C

Reward (30 seconds after, when water started to boil): +1 for the fact that water boiled in the end, but -0.1 reward for each second it took for the water to start boiling. So total reward was -1.9 (-2.9 because the water did not boil for 29 seconds, then +1 for water boiling on 30th second)

Partially Observable Markov Decision Process

The agent does not have all information regarding the current state, and has only an observation, which is a subset of all the information in a given state. Therefore, it is impossible for the agent to truly behave optimally because of lack of information. One way to solve this is to use belief states, or to use RNNs to try to remember previous states to make better judgements on future actions (i.e., we may need states from the previous 10 time steps to know exactly what's going on currently).

Example: We are in a room that is pitch black. At any time instant, we do not know exactly where we are, and if we only take our current state, we would have no idea. However, if we remember that we took 3 steps forward already, we have a much better idea of where we are.

Are my above explanations correct? And if so, isn't it possible to also have Partially Observable Semi Markov Decision Processes?

I just wanted to confirm that my understanding of the different Markov Decision Processes are correct, because they are the fundamentals of reinforcement learning. Also, I read a few literature sources, and some are not consistent with each other. What makes the most sense to me is listed below.

Markov Decision Process

All the states of the environment are known, so the agent has all the information required to make the optimal decision in every state. We can basically assume that the current state has all information about the current state and all the previous states (i.e., the Markov property)

Semi Markov Decision Process

The agent has enough information to make decisions based on the current state. However, the actions of the agent may take a long time to complete, and may not be completed in the next time step. Therefore, the feedback and learning portion should wait until the action is completed before being evaluated. Because the action takes many time steps, "mini rewards" obtained from those time steps should also be summed up.

Example: Boiling water

• State 1: water is at 23 °C
• Action 1: agent sets the stovetop at 200 °C
• Reward (30 seconds after, when water started to boil):
  • +1 for the fact that water boiled in the end, but -0.1 reward for each second it took for the water to start boiling.
  • So, the total reward was -1.9 (-2.9 because the water did not boil for 29 seconds, then +1 for water boiling on 30th second)

Partially Observable Markov Decision Process

The agent does not have all information regarding the current state, and has only an observation, which is a subset of all the information in a given state. Therefore, it is impossible for the agent to truly behave optimally because of a lack of information. One way to solve this is to use belief states, or to use RNNs to try to remember previous states to make better judgements on future actions (i.e., we may need states from the previous 10 time steps to know exactly what's going on currently).

Example

We are in a room that is pitch black. At any time instant, we do not know exactly where we are, and if we only take our current state, we would have no idea. However, if we remember that we took 3 steps forward already, we have a much better idea of where we are.

Are my above explanations correct? And if so, isn't it possible to also have Partially Observable Semi Markov Decision Processes?

I just wanted to confirm that my understanding of the different Markov Decision Processes are correct because they are the fundamentals of reinforcement learning. Also, I read a few literature sources, and some are not consistent with each other. What makes most sense to me is listed below:

Markov Decision Process

All the states of the environment is known so the agent has all the information required to make the optimal decision in every state. We can basically assume that the current state has all information about the current state and all the previous states (i.e., the Markov property)

Semi Markov Decision Process

The Agent has enough information to make decisions based on the current state. However, the agents actions may take a long time to complete, and may not be completed in the next time step. Therefore, the feedback and learning portion should wait until the the action is completed before being evaluated. Because the action takes many time steps, "mini rewards" obtained from those time steps should also be summed up.

Example: Boiling water.

State 1: Water is at 23 C

Action 1: Agent sets stove top at 200 C

Reward (30 seconds after, when water started to boil): +1 for the fact that water boiled in the end, but -0.1 reward for each second it took for the water to start boiling. So total reward was -1.9 (-2.9 because the water did not boil for 29 seconds, then +1 for water boiling on 30th second)

Partially Observable Markov Decision Process

The agent does not have all information regarding the current state, and has only an observation, which is a subset of all the information in a given state. Therefore, it is impossible for the agent to truly behave optimally because of lack of information. One way to solve this is to use belief states, or to use RNNs to try to remember previous states to make better judgements on future actions (i.e., we may need states from the previous 10 time steps to know exactly what's going on currently).

Example: We are in a room that is pitch black. At any time instant, we do not know exactly where we are, and if we only take our current state, we would have no idea. However, if we remember that we took 3 steps forward already, we have a much better idea of where we are.

Are my above explanations correct? And if so, isn't it possible to also have Partially Observable Semi Markov Decision Processes?

I just wanted to confirm that my understanding of the different Markov Decision Processes are correct, because they are the fundamentals of reinforcement learning. Also, I read a few literature sources, and some are not consistent with each other. What makes most sense to me is listed below:

Markov Decision Process

All the states of the environment is known so the agent has all the information required to make the optimal decision in every state. We can basically assume that the current state has all information about the current state and all the previous states (i.e., the Markov property)

Semi Markov Decision Process

The agent has enough information to make decisions based on the current state. However, the actions of the agent may take a long time to complete, and may not be completed in the next time step. Therefore, the feedback and learning portion should wait until the the action is completed before being evaluated. Because the action takes many time steps, "mini rewards" obtained from those time steps should also be summed up.

Example: Boiling water.

State 1: Water is at 23 C

Action 1: Agent sets stove top at 200 C

Reward (30 seconds after, when water started to boil): +1 for the fact that water boiled in the end, but -0.1 reward for each second it took for the water to start boiling. So total reward was -1.9 (-2.9 because the water did not boil for 29 seconds, then +1 for water boiling on 30th second)

Partially Observable Markov Decision Process

The agent does not have all information regarding the current state, and has only an observation, which is a subset of all the information in a given state. Therefore, it is impossible for the agent to truly behave optimally because of lack of information. One way to solve this is to use belief states, or to use RNNs to try to remember previous states to make better judgements on future actions (i.e., we may need states from the previous 10 time steps to know exactly what's going on currently).

Example: We are in a room that is pitch black. At any time instant, we do not know exactly where we are, and if we only take our current state, we would have no idea. However, if we remember that we took 3 steps forward already, we have a much better idea of where we are.

Are my above explanations correct? And if so, isn't it possible to also have Partially Observable Semi Markov Decision Processes?