Questions tagged [markov-chain]
For questions about the use of Markov models in the field of AI/ML.
13
questions
2
votes
0answers
31 views
Are there any ways to model markov chains from time series data?
I want to make a thing that produces a stochastic process from time series data.
The time series data is recorded every hour over the year, which means 24-hour of patterns exist for 365 days.
What I ...
1
vote
0answers
13 views
Predicting Hot Categories In a Reference Manager
Reference managers like Zotero or Mendeley allow researchers to categorize papers into hierarchical categories called collections. The User navigates through a listing of these collections when filing ...
1
vote
1answer
30 views
Can HMM, MRF, or CRF be used to classify the state of a single observation, not the entire observation sequence?
I learn that the Viterbi algorithm used for Hidden Markov Model (HMM) can classify a sequence of hidden states from the corresponding observations; Markov Random Field (MRF) and Conditional Random ...
3
votes
2answers
195 views
Difference in continuing and episodic cases in Sutton and Barto - Introduction to RL, exercise 3.5
Excercise 3.5 The equastions in Section 3.1 are for the continuing
case and need to be modified (very slightly) to apply to episodic
tasks. Show that you know the modifications needed by giving ...
2
votes
1answer
429 views
When is Markov Decision Process (MDP) not adequate for goal-directed learning tasks
In the book Reinforcement Learning: An Introduction (Sutton & Barto, 2018). The authors ask
Exercise 3.2: Is the MDP framework adequate to usefully represent all goal-directed learning tasks? ...
1
vote
1answer
45 views
MDP model for binary search
a number has randomly been chosen from 1 to 3. in each step we can make a guess and we will be told if our guess is equal, bigger or smaller than the chosen number. we're trying to find the number ...
5
votes
1answer
190 views
Detect patterns in sequences of actions
I have to analyse sequences of actions that look more or less like this JSON blob. The question I'm trying to answer is whether there are recurring (sub)patterns that different users adopt when asked ...
2
votes
2answers
294 views
Hidden Markov Model application
I'm very new to the field of AI and HMMs. My question is can HMMs be used to model any time series data? Or does the data have to be that of a Markov process?
In HTK documentation, I see that the ...
1
vote
1answer
183 views
Markov Model for a Traffic Intersection
I need some help in developing a Markov Model for a crossroads there is no one way road and i am assuming at this time that traffic is only allowed to go straight no turns are allowed. There are 4 ...
2
votes
2answers
125 views
Can an Markov decision process be dependent on the past?
As far as I know MDP are independent from the past. But the definition says that the same policy should always take the same action depending on the state.
What if I define my state as the current "...
5
votes
1answer
115 views
How to fill in missing transitions when sampling an MDP transition table?
I have a simulator modelling a relatively complex scenario. I extract ~12 discrete features from the simulator state which forms the basis for my MDP state space.
Suppose I am estimating the ...
3
votes
1answer
167 views
How q-learning solves the issue with value iteration in model-free settings
I can't understand what is the problem in applying value-iteration in reinforcement learning setting (where we don't the reward and transition probabilities). In one of the lectures, the guy said it ...
7
votes
2answers
2k views
What is a Markov chain and how can it be used in creating artificial intelligence?
I believe a Markov chain is a sequence of events where each subsequent event depends probabilistically on the current event. What are examples of the application of a Markov chain and can it be used ...
