3
$\begingroup$

According to the book "Artificial Intelligence: A Modern Approach", "In a known environment, the outcomes (or outcome probabilities if the environment is stochastic) for all actions are given.", and in a deterministic environment, "the next state of the environment is completely determined by the current state and the action executed by the agent...".

What's the difference between the two terms? Don't they mean the same thing?

$\endgroup$
2
$\begingroup$

What's the difference between the two terms? Don't they mean the same thing?

They mean different things, and can occur in any combination.

A known, deterministic environment

This is an environment where the researcher knows how to calculate all the transitions in advance of observing them, and the transition from state $s$ given action $a$ is always to the same next state $s'$ with the same reward $r$.

Example: Any classic board game against a fixed opponent (where any situation allows choice of action, the opponent always picks the same choice in the same situation)

An unknown, deterministic environment

This is an environment where the researcher does not have the knowledge to calculate all the transitions in advance of observing them, but any observation the transition from state $s$ given action $a$ is always to the same next state $s'$ with the same reward $r$.

Example: Simple mechanical physics environments, where initial measurements are unknown, imprecise or the researcher does not want to code knowledge of them into the agent. E.g. pole balancing.

A known, stochastic environment

This is an environment where the researcher knows all the rules about transitions, but those rules include transitions with random elements. Transition from state $s$ given action $a$ varies according to some probability function $p(s'|s,a)$ possibly so does the reward $p(r|s,a)$ - sometimes combined into joint probability function $p(r, s'|s,a)$.

Example: Any board game involving dice, e.g. Backgammon.

An unknown, stochastic environment

This is an environment where the researcher does not know all the rules, or can only calculate expected results with difficulty or to a low level of precision. Transition from state $s$ given action $a$ varies according to some unknown probability function $p(r, s'|s,a)$. Learning the transition function may require many samples of $s, a$, and this will be a statistical approximation.

Example: In practice, most complex environments, including real-world physics with friction, fluids, non-perfect measurements.

It is quite common for purpose of experiments to have a simulation where the environment is technically known (because the researcher wrote it, or has access to the code and underlying models), but that agents are written to treat it like this last, more challenging case. Agents that can figure out how to act without prior knowledge of the environment are often of interest.

| improve this answer | |
$\endgroup$
  • $\begingroup$ so, for example - chess are deterministic, while games with probability are stochastic? $\endgroup$ – Piotr Żak Jan 31 at 7:05
  • $\begingroup$ @PiotrŻak: Yes. Backgammon is a classic board game that is stochastic, due to probilities introduced by the dice. $\endgroup$ – Neil Slater Jan 31 at 7:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.