Newest 'markov-decision-process' Questions

1

vote

0answers

18 views

Why is this Monte Carlo approach scalable for a growing number of states variables and action variables?

I am reading a research paper on the formulation of MDP problems to ICU treatment decision making: Treatment Recommendation in Critical Care: A Scalable and Interpretable Approach in Partially ...

asked 15 hours ago

calveeen

1435 bronze badges

4

votes

0answers

42 views

Interesting examples of discrete stochastic games

SGs are a generalization of MDPs to multiple agents. Like this previous question on MDPs, are there any interesting examples of zero-sum, discrete SGs—preferably with small state and action spaces? I'...

game-theory environment markov-decision-process benchmarks

asked Dec 23 '19 at 20:12

user76284

2731 silver badge7 bronze badges

0

votes

0answers

27 views

What is the difference between value iteration and policy iteration? [duplicate]

In reinforcement learning, what is the difference between policy iteration and value iteration? As much as I understand, in value iteration, you use the Bellman equation to solve for the optimal ...

reinforcement-learning markov-decision-process value-iteration policy-iteration

asked Dec 1 '19 at 11:52

Sheri

1235 bronze badges

0

votes

0answers

22 views

Bandits with missing contexts

Say I learn an optimal policy $\pi(a|c)$ for a contextual multi-armed bandit problem, where the context c is a composite of multiple context variables $c = c_1, c_2, c_3$. For example, the context is ...

reinforcement-learning markov-decision-process

asked Nov 26 '19 at 21:54

user31663

1

vote

0answers

16 views

Finding total number of states in a POMDP

I've been working on a question that is posed in a document I've been reading, that models qualifying for a job as a POMDP. In this model, a person takes 3 exams, and must pass all of them in order to ...

reinforcement-learning markov-decision-process pomdp

asked Nov 19 '19 at 20:10

zay

111 bronze badge

4

votes

1answer

66 views

How to assign rewards in a non-Markovian environment?

I am quite new to the Reinforcement Learning domain and I am curious about something. It seems to be the case that the majority of current research assumes Markovian environments, that is, future ...

reinforcement-learning rewards markov-decision-process environment markov-property

asked Oct 31 '19 at 16:15

thulungair

433 bronze badges

2

votes

1answer

67 views

Solving a Multi-Armed, “Multi-Bandit” Problem

This is the problem: I have 66 slot-machines and for each of them I have 7 possible actions/arms to choose from. At each trial, I have to choose one of 7 actions for each and every one of the 66 slots....

reinforcement-learning policy-gradients markov-decision-process

asked Oct 29 '19 at 1:23

FS93

855 bronze badges

2

votes

1answer

27 views

Why feed actions in later layer in Q network?

I read the DDPG paper, in which the authors state that the actions are fed only later to their Q network: Actions were not included until the 2nd hidden layer of Q. (Sec 7, Experiment Details) So ...

neural-networks reinforcement-learning q-learning markov-decision-process ddpg

asked Oct 22 '19 at 13:06

CGFoX

1865 bronze badges

1

vote

0answers

42 views

What is the state of the art AI training technique for imperfect information 2 player turn based games?

As far as I can tell (correct me if I'm wrong), Alphazero (with MCTS and neural network heuristic function RL) is the state of the art training method for turn based, deterministic, perfect ...

ai-design training game-theory markov-decision-process imperfect-information

asked Oct 3 '19 at 20:08

Bridgeburners

1114 bronze badges

2

votes

1answer

56 views

Markov property in maze solving problem in reinforcement learning

By definition, every state in RL has Markov property, which means that the future state depends only on the current state, not the past states. However, I saw that in some case we can define a state ...

machine-learning reinforcement-learning markov-decision-process

asked Oct 2 '19 at 19:42

Hunnam

1774 bronze badges

3

votes

1answer

38 views

Why does having a fixed policy change a Markov Decision Process (MDP) to a Markov Reward Process (MRP)?

If a policy is fixed, it is said that an MDP becomes an MRP. Why is this so? Aren't the transitions and rewards still parameterized by the action and current state? In other words, aren't the ...

reinforcement-learning markov-decision-process policy

asked Sep 30 '19 at 2:15

Peter

333 bronze badges

5

votes

1answer

34 views

Is the agent aware of a possible different set of actions for each state?

I have a use case where the set of actions is different for different states. Is the agent aware of what actions are valid for which states or is the agent only aware of the entire action space (in ...

reinforcement-learning markov-decision-process

asked Sep 24 '19 at 16:08

Francis Chang

611 bronze badge

2

votes

1answer

61 views

Can someone please help me validate my MDP?

Problem Statement : I have a system with four states - S1 through S4 where S1 is the beginning state and S4 is the end/terminal state. The next state is always better than the previous state i.e if ...

reinforcement-learning rewards policy markov-decision-process

asked Sep 5 '19 at 15:08

Bhavana

436 bronze badges

0

votes

0answers

33 views

Finding optimal Value function and Policy for an MDP

I am solving an RL MDP problem which is model based. I have an MDP which has four possible states S1-S4 and four different actions A1-A4, with S4 being terminal state and S1 is the beginning state. ...

reinforcement-learning rewards policy markov-decision-process

asked Sep 4 '19 at 10:52

Bhavana

436 bronze badges

6

votes

1answer

113 views

Benchmarks for reinforcement learning in discrete MDPs

To compare the performance of various algorithms for perfect information games, reasonable benchmarks include reversi and m,n,k-games (generalized tic-tac-toe). For imperfect information games, ...

reinforcement-learning environment markov-decision-process benchmarks

asked Sep 1 '19 at 18:11

user76284

2731 silver badge7 bronze badges

2

votes

1answer

43 views

Can I have different rewards for a single action based on which state it transitions to?

I am working on an MDP where there are four states and ten actions. I am supposed to derive the optimal policy to reach the desired state. At any state, a particular action can take you to any of the ...

reinforcement-learning markov-decision-process rewards value-iteration

asked Aug 30 '19 at 12:48

Bhavana

436 bronze badges

0

votes

0answers

13 views

Derivation for Value Iteration of CVaR

I am reading a paper named Risk-sensitive and Robust Decision-making: a CVaR Optimization Approach. In appendix A.3 they provide a proof for their Theorem $4$. The $n=1$ case for equation (11) is ...

reinforcement-learning optimization markov-decision-process papers

asked Jul 25 '19 at 2:02

altria

1

11

votes

1answer

290 views

How to stay a up-to-date researcher in ML/RL community?

As a student who wants to work on machine learning, I would like to know how it is possible to start my studies and how to follow it to stay up-to-date. For example, I am willing to work on RL and MAB ...

machine-learning reinforcement-learning research markov-decision-process

asked Jul 18 '19 at 11:54

Katatonia

4512 silver badges7 bronze badges

4

votes

1answer

36 views

Why are state transitions in MDPs probabilistic rather than deterministic?

I've read that for MDPs the state transition function $P_a(s, s')$ is a probability. This seems strange to me for modeling because most environments (like video games) are deterministic. Now, I'd ...

machine-learning reinforcement-learning markov-decision-process

asked Jun 17 '19 at 6:38

personjerry

1415 bronze badges

5

votes

2answers

230 views

Why am I getting the incorrect value of lambda?

I am trying to solve for lambda using Temporal Difference Learning I am trying to figure out what lambda I need, to make TD(λ)=TD(1) but I get the incorrect value of lambda. Here is how I did: <...

reinforcement-learning python markov-decision-process temporal-difference

asked May 20 '19 at 5:58

Amanda

2054 bronze badges

5

votes

2answers

53 views

Reinforcement learning with uniformly random dynamics

Suppose I have an MDP $(S, A, p, R)$ where the $p(s_j|s_i,a_i)$ is uniform, i.e given an state $s_i$ and an action $a_i$ all states $s_j$ are equally probable. Now I want to find an optimal policy ...

reinforcement-learning markov-decision-process

asked May 4 '19 at 23:40

grok

1513 bronze badges

2

votes

1answer

34 views

Are successive actions independent?

The proof of the consistency of the per-decision importance sampling estimator assumes the independence of $$\frac{\pi(A_t|S_t)}{b(A_t|S_t)}R_{t+1}\quad\text{ and }\quad \prod_{k=t+1}^{T-1}\frac{\pi(...

reinforcement-learning markov-decision-process importance-sampling

asked Apr 14 '19 at 22:19

Philip Raeisghasem

1,3771 silver badge24 bronze badges

2

votes

0answers

36 views

Does everyone still use discount rates?

In Section 10.4 of Sutton and Barto's RL book, they argue that the discount rate $\gamma$ has no effect in continuing settings. They show (at least for one objective function) that the average of the ...

reinforcement-learning discount-factor markov-decision-process

asked Apr 4 '19 at 4:02

Philip Raeisghasem

1,3771 silver badge24 bronze badges

3

votes

2answers

619 views

Can Q-learning be used in a POMDP?

Can Q-learning (and SARSA) be directly used in a Partially Observable Markov Decision Process (POMDP)? If not, why not? My intuition is that the policies learned will be terrible because of partial ...

reinforcement-learning q-learning pomdp markov-decision-process sarsa

asked Apr 3 '19 at 2:40

drerD

2361 silver badge3 bronze badges

2

votes

1answer

42 views

Do we have to consider the feasability of an action when defining the reward function of a MDP?

Do we have to consider if (s is given) an action a can lead to s' when defining a reward function? To be more specific: Let's say I have a 1D Map like: |A|B|C|D| ...

rewards markov-decision-process

asked Mar 22 '19 at 20:42

Bryan McGill

3131 silver badge9 bronze badges

5

votes

1answer

733 views

How can we use linear programming to solve an MDP?

Apparently, we can solve an MDP (that is, we can find the optimal policy for a given MDP) using a linear programming formulation. What's the basic idea behind this approach? I think you should start ...

reinforcement-learning optimization markov-decision-process linear-programming

asked Mar 14 '19 at 21:12

nbro

10.9k5 gold badges23 silver badges57 bronze badges

3

votes

0answers

70 views

What is a generalized MDP?

What is a generalized MDP? How is it different than a "regular" MDP? How does it generalise the notion of an MDP? Why do we need a generalised MDP? Do generalised MDPs have some practical usefulness ...

reinforcement-learning definitions markov-decision-process

asked Mar 14 '19 at 20:47

nbro

10.9k5 gold badges23 silver badges57 bronze badges

1

vote

1answer

149 views

Unable to understand the second iteration update in value iteration algorithm for solving MDP

I am trying to understand the value iteration method for Markov Decision Process(MDP) and I was referring ot UC Berkeley's slides titled Markov Decision Processes and Exact Solution Methods On slide ...

reinforcement-learning ai-basics value-iteration markov-decision-process

asked Mar 12 '19 at 21:09

Devendra Vyas

132 bronze badges

2

votes

1answer

63 views

Does the observation function for POMDP always add up to 1?

I was reading in the article A tutorial on partially observable Markov decision processes (p. 120), by Michael L. Littman, that $\sum_{z \in Z}O(a, s',z) =1$, where $a$ is the action, $s'$ the next ...

reinforcement-learning markov-decision-process pomdp

asked Mar 8 '19 at 21:21

Bryan McGill

3131 silver badge9 bronze badges

3

votes

2answers

211 views

Difference in continuing and episodic cases in Sutton and Barto - Introduction to RL, exercise 3.5

Excercise 3.5 The equastions in Section 3.1 are for the continuing case and need to be modified (very slightly) to apply to episodic tasks. Show that you know the modifications needed by giving ...

reinforcement-learning markov-decision-process rl-an-introduction markov-chain

asked Mar 7 '19 at 15:52

Jakub Bielan

654 bronze badges

5

votes

1answer

86 views

Should I model my problem as a semi-MDP?

I have a system (like a bank) that people (customers) are entered into the systems by a Poisson process, so the time between the arrival of people (two consecutive customers) will be a random variable....

reinforcement-learning markov-decision-process semi-mdp hierarchical-rl

asked Feb 17 '19 at 12:29

Katatonia

4512 silver badges7 bronze badges

0

votes

0answers

107 views

How can we estimate the transition model and reward function?

In reinforcement learning (RL), there are model-based and model-free algorithms. In short, model-based algorithms use a transition model (e.g. a probability distribution) and the reward function, even ...

reinforcement-learning algorithm markov-decision-process model-based model-free

asked Feb 16 '19 at 18:59

nbro

10.9k5 gold badges23 silver badges57 bronze badges

3

votes

3answers

216 views

Can the rewards be stochastic when the transition model is deterministic?

Suppose we have a deterministic environment where knowing $s,a$ determines $s'$. Is it possible to get two different rewards $r\neq r'$ in some state $s_{\text{fixed}}$? Assume that $s_{\text{fixed}}$ ...

reinforcement-learning markov-decision-process

asked Feb 12 '19 at 10:33

rewardbabe

311 bronze badge

1

vote

1answer

55 views

Is the next state drawn from the joint distribution of the previous state and action?

Suppose $G_t$, the discounted return at time $t$ is defined as: $$ G_t \triangleq R_t+\gamma R_{t+1}+\gamma^{2}R_{t+2} + \cdots = \sum_{j=1}^{\infty} \gamma^{k}R_{t+k}$$ where $R_t$ is the reward at ...

reinforcement-learning math markov-decision-process

asked Feb 12 '19 at 5:15

reinforcemenguy12

191 bronze badge

2

votes

1answer

70 views

What is the relation between a policy which is the solution to a MDP and a policy like $\epsilon$-greedy?

In the context of reinforcement learning, a policy, $\pi$, is often defined as a function from the space of states, $\mathcal{S}$, to the space of actions, $\mathcal{A}$, that is, $\pi : \mathcal{S} \...

reinforcement-learning terminology definitions markov-decision-process policy

asked Feb 10 '19 at 16:34

nbro

10.9k5 gold badges23 silver badges57 bronze badges

4

votes

2answers

71 views

How are the reward functions $R(s)$, $R(s, a)$ and $R(s, a, s')$ equivalent?

In this video, the lecturer states that $R(s)$, $R(s, a)$ and $R(s, a, s')$ are equivalent representations of the reward function. Intuitively, this is the case, according to the same lecturer, ...

reinforcement-learning definitions markov-decision-process proofs

asked Feb 7 '19 at 15:38

nbro

10.9k5 gold badges23 silver badges57 bronze badges

1

vote

0answers

41 views

What limitations does the Markov property place on real time learning?

The Markov property is the dependence of a system's future state probability distribution solely on the present state, excluding any dependence on past system history. The presence of the Markov ...

reinforcement-learning markov-decision-process markov-property

asked Jan 12 '19 at 0:29

Douglas Daseeco

7,2411 gold badge15 silver badges50 bronze badges

1

vote

2answers

315 views

Is Monte Carlo Tree Search appropriate for problems with large state and action spaces?

I'm doing a research on a finite-horizon Markov decision process with $t=1, \dots, 40$ periods. In every time step $t$, the (only) agent has to chose an action $a(t) \in A(t)$, while the agent is in ...

reinforcement-learning monte-carlo-tree-search markov-decision-process finite-markov-decision-process

asked Jan 9 '19 at 12:25

D. B.

111 bronze badge

5

votes

1answer

84 views

What is the appropriate approach to playing a game with incomplete state information?

I have a steady hex-map and turn-based war game featuring WWII carrier battles. I would like to improve the fixed policy for the AI using reinforcement learning. I have some beginner's questions, ...

reinforcement-learning game-ai markov-decision-process pomdp

asked Dec 23 '18 at 10:44

Carrier Battles

1115 bronze badges

1

vote

1answer

144 views

Importance Sampling Ratio Probability

When reading Reinforcement Learning by Sutton and Barto, I came across the importance sampling ratio. The first equation, I believe, describes the probability a particular sequence is obtained given ...

reinforcement-learning markov-decision-process

asked Dec 17 '18 at 14:20

Zach Zhao

132 bronze badges

1

vote

0answers

40 views

How to generalize finite MDP to general MDP?

Suppose, for simplicity sake, to be in a discrete time domain with the action set being the same for all states $S \in \mathcal{S}$. Thus, in a finite Markov Decision Process, the sets $\mathcal{A}$, $...

markov-decision-process finite-markov-decision-process

asked Nov 23 '18 at 13:25

gvgramazio

4363 silver badges16 bronze badges

1

vote

0answers

383 views

Is my understanding of the differences between MDP, Semi MDP and POMDP correct?

I just wanted to confirm that my understanding of the different Markov Decision Processes are correct, because they are the fundamentals of reinforcement learning. Also, I read a few literature ...

reinforcement-learning definitions markov-decision-process pomdp semi-mdp

asked Oct 29 '18 at 21:33

Rui Nian

3551 silver badge10 bronze badges

3

votes

2answers

1k views

What is a time-step in a Markov Decision Process?

The “Discounted sum of future rewards” using discount factor $\gamma$ is $\gamma$ (reward in 1 time step) + $\gamma^2$ (reward in 2 time steps) + $\gamma^3$ (reward in 3 time steps) + ... I am ...

reinforcement-learning markov-decision-process

asked Oct 27 '16 at 19:57

Abhishek Bhatia

3343 silver badges14 bronze badges

Questions tagged [markov-decision-process]

Related Tags