9

A Markov model includes the probability of transitioning to each state considering the current state. "Each state" may be just one point - whether it rained on specific day, for instance - or it might look like multiple things - like a pair of words. You've probably seen automatically generated weird text that almost makes sense, like Garkov (the output of a ...


5

You wouldn't, normally. A HMM is used to model sequences of observations, and it would not make sense to use it for image recognition. Unless they are sequential, such as strokes in handwriting. HMMs are typically used in fields such as speech recognition and part-of-speech tagging. Here you observe a sequence of events that you want to fit to a model in ...


4

It's not possible, as in the chain illustrated there are no transitions between A and C, C and D, and D and F. Only sequences where transitions exist are possible. The choice of transitions is arbitrary; it depends on what you want to model with it. As DuttaA says in his comment, you can imagine that all nodes are linked with all other nodes, but those ...


3

For normal value iteration, you need to have the model, i.e. the transition probability, denoted by $P(s' \mid s,a)$. With Q-learning, you use the current reward and the already stored Q value: The relation between the value function $V(s)$ and the Q function $Q(s, a)$ is that the $V(s)$ function is simply the value of the action $a$, such that $Q(s, a)$ is ...


3

It's not totally clear from your description, but it sounds like you may be onto something like an Additive Markov Chain.


2

Is it just about final states? So for $s \in S$ when S is not final? You are thinking the right way, but to represent what you mean you don't need to write out "when $s$ is not final" - although that would be fine (and is used in some places), there is a more concise way of saying that given to you by the book. As this is a formal exercise from the book, ...


2

Why not use a single agent to control the intersection with the following rules: define 8 traffic lights Each light has two possible values, 0 or 1, with 0 = Red and 1 = Green Lights are as follows: 0 = North to South 1 = North to East (Left Turn) 2 = East to West 3 = East to South (Left Turn) 4 = South to North 5 = South to West (Left Turn) 6 = West to ...


1

I am not an expert on this, but I'll try to explain my understnding of this. A Bayesian Network is a Directed Graphical Model (DGM) with the ordered Markov property i.e the relationship of a node (random variable) depends only on its immediate parents and not its predecessors (generalized from first order Markov process). A Markov chain on the other hand ...


1

The main difference between a Bayesian network and a Markov chain is not that a Markov Chain is not directional, it is that the graph of the Bayesian network is not trivial whereas the graph of a Markov chain would be somewhat trivial, as all the previous $k$ nodes would just point to the current node. To illustrate further why this would be trivial, we let ...


1

MPD are not independent from the past, but future actions starting from current state are independent from the past i.e. the probability of the next state given all previous states is the same as the probability of the next state given the previous state. Any state representation which is composed by the full history is a MDP, because looking at the history (...


1

This task falls within the overlapping fields of information extraction and pattern mining. Information extraction involves automatically extracting instances of specified relations from data. While pattern mining involves using data mining algorithms to discover interesting, unexpected and useful patterns between data in databases (Philippe F). On your ...


1

I assume you use the 12 discrete features as state variables, and for each of these variables you will have at least two values. So the minimum number of states will be: 2^12 = 4096, which gives (2^12)^2 = 16777216 possible transitions. In order to reach this you will need a huge amount of simulations, also taking into account that this number is a minimum ...


1

(this was intended as a comment, but turned out long and longer) A couple of points to elaborate on Ben's answer: It is possible to generate different models (out of existing data!) and then look for the model that best fit new data (e.g. with knn). Example: States = {sleep, eat, walk, work} Model 1: Most probable sequence on weekdays, say: sleep → sleep ...


Only top voted, non community-wiki answers of a minimum length are eligible