# Tag Info

Accepted

### What is a Markov chain and how can it be used in creating artificial intelligence?

A Markov model includes the probability of transitioning to each state considering the current state. "Each state" may be just one point - whether it rained on specific day, for instance - or it might ...
• 2,609
Accepted

### What is ergodicity in a Markov Decision Process (MDP)?

In short, the relevant class of a MDPs that guarantees the existence of a unique stationary state distribution for every deterministic stationary policy are unichain MDPs (Puterman 1994, Sect. 8.3). ...
• 579
Accepted

### How can I use a Hidden Markov Model to recognize images?

You wouldn't, normally. A HMM is used to model sequences of observations, and it would not make sense to use it for image recognition. Unless they are sequential, such as strokes in handwriting. HMMs ...
• 5,397
Accepted

### In the Markov chain, how are the directions to each successive state defined?

It's not possible, as in the chain illustrated there are no transitions between A and C, C and D, and D and F. Only sequences where transitions exist are possible. The choice of transitions is ...
• 5,397
Accepted

### Can an Markov decision process be dependent on the past?

MPD are not independent from the past, but future actions starting from current state are independent from the past i.e. the probability of the next state given all previous states is the same as the ...

### Can an Markov decision process be dependent on the past?

It's not totally clear from your description, but it sounds like you may be onto something like an Additive Markov Chain.
• 3,767
Accepted

### Difference in continuing and episodic cases in Sutton and Barto - Introduction to RL, exercise 3.5

Is it just about final states? So for $s \in S$ when S is not final? You are thinking the right way, but to represent what you mean you don't need to write out "when $s$ is not final" - although ...
• 32.7k

### Detect patterns in sequences of actions

This task falls within the overlapping fields of information extraction and pattern mining. Information extraction involves automatically extracting instances of specified relations from data. While ...
• 1,186

### Markov Model for a Traffic Intersection

Why not use a single agent to control the intersection with the following rules: define 8 traffic lights Each light has two possible values, 0 or 1, with 0 = Red and 1 = Green Lights are as follows: ...
• 251

### What is a Markov chain and how can it be used in creating artificial intelligence?

(this was intended as a comment, but turned out long and longer) A couple of points to elaborate on Ben's answer: It is possible to generate different models (out of existing data!) and then look ...
• 538
1 vote
Accepted

### Forward Diffusion Process Derivation In Diffusion Models

The relationship between $x_t$ and $x_{t-1}$ is as follows: $$x_t = \sqrt{1-\beta_t}x_{t-1}+\sqrt{\beta_t}\epsilon_t,\quad \epsilon_t\sim\mathcal{N}(0,I).$$ Not only is a small amount of noise added,...
1 vote

### What is the difference between a Bayesian Network and a Markov Chain?

The main difference between a Bayesian network and a Markov chain is not that a Markov Chain is not directional, it is that the graph of the Bayesian network is not trivial whereas the graph of a ...
• 4,920
1 vote

### How is the probability transition matrix populated in the Markov process (chain) for a board game?

Transition probability matrix cannot be initialized. Your game world has some rules. Since we do not know these rules we can approximate them. To do this, we should run the game over and over and ...
• 957

Only top scored, non community-wiki answers of a minimum length are eligible