Questions tagged [markov-chain]

For questions about the use of Markov models in the field of AI/ML.

Filter by
Sorted by
Tagged with
2
votes
0answers
34 views

Can $Q$-learning or SARSA be thought of a Markov Chain?

I might just be overthinking a very simple question but nonetheless the following has been bugging me a lot. Given an MDP with non-trivial state and action sets, we can implement the SARSA algorithm ...
1
vote
1answer
75 views

Can HMM, MRF, or CRF be used to classify the state of a single observation, not the entire observation sequence?

I learn that the Viterbi algorithm used for Hidden Markov Model (HMM) can classify a sequence of hidden states from the corresponding observations; Markov Random Field (MRF) and Conditional Random ...
0
votes
0answers
10 views

Finding Cycles in a State Sequence

Suppose I observe a set of states $\mathbf{X} = \{X_{1}, X_{2}, \ldots, X_{K}\}$ over time. I assume that there exist $M$ cycles $\mathbf{C} = \{C_{1}, C_{2}, \ldots, C_{M} \}$ in the observed state ...
1
vote
1answer
55 views

How is the probability transition matrix populated in the Markov process (chain) for a board game?

Following on from my other (answered) question: With regards to the Markov process (chain), if an environment is a board game and its states are the various position the game pieces may be in, how ...
6
votes
2answers
132 views

What is the difference between a Bayesian Network and a Markov Chain?

I am trying to understand the difference between a Bayesian Network and a Markov Chain. When I search for this one the web, the unanimous solution seems to be that a Bayesian Network is directional (...
3
votes
1answer
56 views

How can I use a Hidden Markov Model to recognize images?

How could I use a 16x16 image as an input in a HMM? And at the same time how would I train it? Can I use backpropagation?
1
vote
1answer
100 views

In the Markov chain, how are the directions to each successive state defined?

I'm watching the David Silver series on YT which has raised a couple of questions: In the Markov process (or chain), how are the directions to each successive state defined? For example, how are the ...
2
votes
0answers
37 views

Are there any ways to model markov chains from time series data?

I want to make a thing that produces a stochastic process from time series data. The time series data is recorded every hour over the year, which means 24-hour of patterns exist for 365 days. What I ...
1
vote
0answers
18 views

Predicting Hot Categories In a Reference Manager

Reference managers like Zotero or Mendeley allow researchers to categorize papers into hierarchical categories called collections. The User navigates through a listing of these collections when filing ...
3
votes
2answers
378 views

Difference in continuing and episodic cases in Sutton and Barto - Introduction to RL, exercise 3.5

Excercise 3.5 The equastions in Section 3.1 are for the continuing case and need to be modified (very slightly) to apply to episodic tasks. Show that you know the modifications needed by giving ...
5
votes
1answer
207 views

Detect patterns in sequences of actions

I have to analyse sequences of actions that look more or less like this JSON blob. The question I'm trying to answer is whether there are recurring (sub)patterns that different users adopt when asked ...
2
votes
2answers
161 views

Can an Markov decision process be dependent on the past?

As far as I know MDP are independent from the past. But the definition says that the same policy should always take the same action depending on the state. What if I define my state as the current "...
1
vote
1answer
189 views

Markov Model for a Traffic Intersection

I need some help in developing a Markov Model for a crossroads there is no one way road and i am assuming at this time that traffic is only allowed to go straight no turns are allowed. There are 4 ...
6
votes
1answer
128 views

How to fill in missing transitions when sampling an MDP transition table?

I have a simulator modelling a relatively complex scenario. I extract ~12 discrete features from the simulator state which forms the basis for my MDP state space. Suppose I am estimating the ...
7
votes
2answers
3k views

What is a Markov chain and how can it be used in creating artificial intelligence?

I believe a Markov chain is a sequence of events where each subsequent event depends probabilistically on the current event. What are examples of the application of a Markov chain and can it be used ...