# Questions tagged [markov-decision-process]

For questions related to the concept of Markov decision process (MDP), which is a mathematical framework for modeling decision making in situations where outcomes are partly random and partly under the control of a decision-maker. The concept of MDP is useful for studying optimization problems solved via dynamic programming and reinforcement learning.

146 questions
Filter by
Sorted by
Tagged with
36 views

### Can RL still learn in a scenario where current state and the next state are independant?

I am trying to implement reinforcement learning into my real-world problem. One thing making me hesitant to apply RL is that this real-world problem of mine is unique in a way how every state is ...
53 views

### Implementation of MDP in python to determine when to take action clean

I am trying to model the following problem as a Markov decision process. In a steel melting shop of a steel plant, iron pipes are used. These pipes generate rust over time. Adding an anti-rusting ...
29 views

### Doubt in calculating the optimal costs and value after n steps of a MDP problem

MDP problem - A server requires information from a sensor. The server would like the information to be fresh. However, there is a cost to querying information from the sensor. Specifically, the state ...
22 views

### Reinforcement Learning for Finite Time Horizon and Non-Trivial Terminal Reward

I notice that most Deep Reinforcement Learning (DRL) works focus on Markov Decision Process (MDP) with an infinite time horizon. Are there any algorithms that work well on finite MDP and non-trivial ...
56 views

### In reinforcement learning, why are policies defined as functions of states and not observations?

I am new to RL and I am following Sutton & Barto's book. My doubt is, when we talk about the policy of our agent, we say it is the probability of taking some action $a$ given the state $s$. ...
29 views

### Discard irrelavant states from a MDP

I came across this question about MDP. From the look of it, it seems the full MDP is reducible if the discarded state only have 1 way in and out but is it really so if we change the discounted factor? ...
16 views

### How does the Markov assumption hold true for episodic task?

The Markov assumption assumes that the current state is sufficient for taking the next action. Consider an episodic task, where the RL agent is trying to learn to play the game of tic-tac-toe. Here, ...
49 views

### Calculating state-value functions in Markov Decision Process

I am watching David Silver's lectures on RL available on YouTube. My question here is with regard to Lecture 2 (Link to Video). At 1:11:00, I could not understand how he is calculating the state-value ...
37 views

### Markov Decision Processes with variable epoch lengths

I am working on modeling a transportation problem as an MDP. Multiple trucks move material from one node to various other nodes in a network. However, the time it takes a truck to travel between any 2 ...
32 views

### Is there any inherent assumption of start and goal states in an MDP?

MDP stands for the Markov decision process. It is a 5-length tuple used in reinforcement learning. $$MDP = (S, A, T, R, \pi)$$ $S$ stands for a set of states, also called state space. $A$ stands for a ...
11 views

### Is it better to model a Contextual Multi-Armed Bandit problem as an MDP with a non-zero discount factor than treating it as it is?

I'd like to ask if it is, generally, better to model a problem that naturally appears as a Contextual Multi-Armed Bandit like Recommender Systems as an Markov Decision Process with a non-zero discount ...
43 views

56 views

### Why do we discount the state distribution?

In Reinforcement Learning, it is common to use a discount factor $\gamma$ to give less importance to future rewards when calculating the returns. I have also seen mention of discounted state ...
61 views

### Given a sequence of states followed by the agent, is it possible to find the Q-value for a state-action pair not in this sequence?

Assume you are given a sequence of states followed by the agent, generated by a random policy, $[s_0, s_1, s_2, \dots, s_n]$. Furthermore, assume the MDP is fully observable and time is discrete. Is ...
40 views

70 views

### How can I model a problem as an MDP if the agent does not follow the successive order of states?

In my problem, the agent does not follow the successive order of states, but selects with $\epsilon$-greedy the best pair (state, action) from a priority queue. More specifically, when my agent goes ...
23 views

### Understanding example for Improved Policy Iteration for POMDPs

I was going through this paper by Hansen. This paper proposes policy improvement by first converting set of $\alpha$ vectors into finite state controller and then comparing them to obtain improved ...
51 views

### When we have multiple traces, do we average over traces or the total number of times we have visited that state?

I am confused about the workings of the first- and every-visit MC. My first question is, when we have multiple traces, do we average over traces or the total number of times we have visited that state?...
### Why is it useful to define the return as the sum of the rewards from time $t$ onward rather than up to $t$?
Why is it useful to define the return as the sum of the rewards from time $t$ onward rather than up to $t$? The return for an MDP is usually defined as $$G_t=R_{t+1}+R_{t+2}+ \dots +R_T$$ Why is this ...