Questions tagged [markov-property]

For questions related to the Markov property or Markov assumption (that is, the assumption that the "future is independent of the past, given the present"), which underlies e.g. most reinforcement learning algorithms.

Filter by
Sorted by
Tagged with
1 vote
1 answer
49 views

How is the Markov property of a general state-space model derived?

Below is the derivation for the Markov property of a general state-space model. The red part is not clear. Could someone please explain the steps in the sequential derivation for the red part?
DSPinfinity's user avatar
0 votes
0 answers
1k views

How to encode time embeddings in a diffusion model for 1D vectors?

For a project I'm working on, I'd like to try a diffusion model like illustrated in the paper by Ho et Al to generate 1D vectors. What I'm trying to figure out at the moment, is what kind of ...
James Arten's user avatar
1 vote
0 answers
167 views

Implementation of MDP in python to determine when to take action clean

I am trying to model the following problem as a Markov decision process. In a steel melting shop of a steel plant, iron pipes are used. These pipes generate rust over time. Adding an anti-rusting ...
shan's user avatar
  • 111
1 vote
1 answer
874 views

What is the difference between environment states and agent states in terms of Markov property?

I'm going through the David Silver RL course on YouTube. He talks about environment internal state $S^e_t$, and agent internal state $S^a_t$. We know that state $s$ is Markov if $$\mathbb{P}\{S_t=s|S_{...
Stanko Kovacevic's user avatar
1 vote
2 answers
2k views

Reinforcement Learning for an environment that is non-markovian [closed]

I will start working on a project where we want to optimize the production of a chemical unit through reinforcement learning approach. From the SME's, we already obtained a simulator code that can ...
chupa_kabra's user avatar
3 votes
1 answer
143 views

Why can we take the action $a$ from the next state $s'$ in the max part of the Q-learning update rule, if that action doesn't lead to any reward?

I'm using OpenAI's cartpole environment. First of all, is this environment not Markov? Knowing that, my main question concerns Q-learning and off-policy methods: For me, there is something weird in ...
JeanMi's user avatar
  • 165
3 votes
2 answers
2k views

Reinforcement Learning algorithm with rewards dependent both on previous action and current action

Problem description: Suppose we have an environment, where a reward at time step $t$ is dependent not only on the current action, but also on previous action in the following way: if current action ==...
FQT's user avatar
  • 33
0 votes
0 answers
229 views

Is my reward function non-Markovian?

I am working on an RL problem where the time when the agent obtains the reward for taking action $a$ in time step $t$ is stochastic. In fact, there is no immediate reward for taking action $a$ in time ...
shirin elahi's user avatar
3 votes
0 answers
306 views

Policy gradient: Does it use the Markov property?

To derive the policy gradient, we start by writing the equation for the probability of a certain trajectory (e.g. see spinningup tutorial): $$ \begin{align} P_\theta(\tau) &= P_\theta(s_0, a_0, ...
Gerges's user avatar
  • 131
2 votes
0 answers
85 views

Can $Q$-learning or SARSA be thought of a Markov Chain?

I might just be overthinking a very simple question but nonetheless the following has been bugging me a lot. Given an MDP with non-trivial state and action sets, we can implement the SARSA algorithm ...
dezdichado's user avatar
5 votes
1 answer
468 views

Why does TD Learning require Markovian domains?

One of my friends and I were discussing the differences between Dynamic Programming, Monte-Carlo, and Temporal Difference (TD) Learning as policy evaluation methods - and we agreed on the fact that ...
stoic-santiago's user avatar
1 vote
0 answers
36 views

What are the most common non-Markov RL paradigms?

I am interested in doing model-free RL but not using the Markov assumptions typical for MDPs or POMDPs. What are alternative paradigms that don't rely on the Markov assumptions? Are there any common ...
Stan Shunpike's user avatar
4 votes
1 answer
240 views

How is the Markovian property consistent in reinforcement learning based scheduling?

In Reinforcement Learning, an MDP model incorporates the Markovian property. A lot of scheduling applications in a lot of disciplines use reinforcement learning (mostly deep RL) to learn scheduling ...
ephemeral's user avatar
  • 143
4 votes
1 answer
86 views

Is a neural network able to optimize itself for speed?

I am experimenting with OpenAI Gym and reinforcement learning. As far as I understood, the environment is waiting for the agent to make a decision, so it's a sequential operation like this: ...
Thomas Weller's user avatar
3 votes
1 answer
3k views

What does the Markov assumption say about the history of state sequences?

Does the Markov assumption say that the conditional probability of the next state only depends on the current state or does it say that the conditional probability depends on a fixed finite number of ...
MScott's user avatar
  • 445
4 votes
1 answer
464 views

How to assign rewards in a non-Markovian environment?

I am quite new to the Reinforcement Learning domain and I am curious about something. It seems to be the case that the majority of current research assumes Markovian environments, that is, future ...
thulungair's user avatar
3 votes
2 answers
253 views

Can non-Markov environments also be deterministic?

The definition of deterministic environment I am familiar with goes as follows: The next state of the agent depends only on the current state and the action chosen by the agent. By exclusion, ...
user9007131's user avatar
2 votes
1 answer
77 views

Is the Markov property assumed in the forward algorithm?

I'm majoring in pure linguistics (not computational), and I don't have any basic knowledge regarding computational science or mathematics. But I happen to take the "Automatic Speech Recognition&...
Jeeyoung Jeon's user avatar
4 votes
0 answers
75 views

What research has been done on learning non-Markovian reward functions?

Recently, some work has been done planning and learning in Non-Markovian Decision Processes, that is, decision-making with temporally extended rewards. In these settings, a particular reward is ...
Gavin Rens's user avatar
1 vote
0 answers
68 views

What limitations does the Markov property place on real time learning?

The Markov property is the dependence of a system's future state probability distribution solely on the present state, excluding any dependence on past system history. The presence of the Markov ...
Douglas Daseeco's user avatar