Skip to main content
10 votes
Accepted

What is a Markov chain and how can it be used in creating artificial intelligence?

A Markov model includes the probability of transitioning to each state considering the current state. "Each state" may be just one point - whether it rained on specific day, for instance - or it might ...
Ben N's user avatar
  • 2,609
7 votes
Accepted

What is ergodicity in a Markov Decision Process (MDP)?

In short, the relevant class of a MDPs that guarantees the existence of a unique stationary state distribution for every deterministic stationary policy are unichain MDPs (Puterman 1994, Sect. 8.3). ...
mikkola's user avatar
  • 579
5 votes
Accepted

How can I use a Hidden Markov Model to recognize images?

You wouldn't, normally. A HMM is used to model sequences of observations, and it would not make sense to use it for image recognition. Unless they are sequential, such as strokes in handwriting. HMMs ...
Oliver Mason's user avatar
  • 5,397
4 votes
Accepted

In the Markov chain, how are the directions to each successive state defined?

It's not possible, as in the chain illustrated there are no transitions between A and C, C and D, and D and F. Only sequences where transitions exist are possible. The choice of transitions is ...
Oliver Mason's user avatar
  • 5,397
3 votes
Accepted

Can an Markov decision process be dependent on the past?

MPD are not independent from the past, but future actions starting from current state are independent from the past i.e. the probability of the next state given all previous states is the same as the ...
Marcelo Fornet's user avatar
3 votes

Can an Markov decision process be dependent on the past?

It's not totally clear from your description, but it sounds like you may be onto something like an Additive Markov Chain.
mindcrime's user avatar
  • 3,767
2 votes
Accepted

Difference in continuing and episodic cases in Sutton and Barto - Introduction to RL, exercise 3.5

Is it just about final states? So for $s \in S$ when S is not final? You are thinking the right way, but to represent what you mean you don't need to write out "when $s$ is not final" - although ...
Neil Slater's user avatar
  • 32.7k
2 votes

Detect patterns in sequences of actions

This task falls within the overlapping fields of information extraction and pattern mining. Information extraction involves automatically extracting instances of specified relations from data. While ...
Seth Simba's user avatar
  • 1,186
2 votes

Markov Model for a Traffic Intersection

Why not use a single agent to control the intersection with the following rules: define 8 traffic lights Each light has two possible values, 0 or 1, with 0 = Red and 1 = Green Lights are as follows: ...
A.Rashad's user avatar
  • 251
2 votes

What is a Markov chain and how can it be used in creating artificial intelligence?

(this was intended as a comment, but turned out long and longer) A couple of points to elaborate on Ben's answer: It is possible to generate different models (out of existing data!) and then look ...
Luis's user avatar
  • 538
1 vote
Accepted

Forward Diffusion Process Derivation In Diffusion Models

The relationship between $x_t$ and $x_{t-1}$ is as follows: $$ x_t = \sqrt{1-\beta_t}x_{t-1}+\sqrt{\beta_t}\epsilon_t,\quad \epsilon_t\sim\mathcal{N}(0,I). $$ Not only is a small amount of noise added,...
Patrick Johnstone's user avatar
1 vote

What is the difference between a Bayesian Network and a Markov Chain?

The main difference between a Bayesian network and a Markov chain is not that a Markov Chain is not directional, it is that the graph of the Bayesian network is not trivial whereas the graph of a ...
David's user avatar
  • 4,920
1 vote

How is the probability transition matrix populated in the Markov process (chain) for a board game?

Transition probability matrix cannot be initialized. Your game world has some rules. Since we do not know these rules we can approximate them. To do this, we should run the game over and over and ...
Aray Karjauv's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible