0
$\begingroup$

Can you describe this system in more detail? I understand that the environment sends a signal indicating whether or not the action taken by the agent was 'good' or not, but it seems too simple. Basically, can you detail the nitty-gritty workings of this system? I dunno, I may just be overthinking things. Sorry if this question was too broad.

$\endgroup$
2
$\begingroup$

In this case, the word "system" refers to a Markov decision process (MDP), which is the mathematical model used to represent the reinforcement learning (RL) problem or, in general, a decision making problem. Recall that, in RL, the problem consists in finding an (optimal) policy, which is a policy that allows the agent to collect the highest amount of reward (in the long run). Hence, in RL, the MDP is the problem and the optimal policy (for that MDP) is the solution.

The MDP is composed of the set of states of the environment $S$, the set of possible actions that the RL agent can take $A$, a transition function, $P(s_{t+1}=s'\mid s_{t}=s, a_t = a)$, which is a probability distribution and describes the dynamics of the environment, and a reward function $R_a(s', s)$, which is a function that describes the "reward system" of the environment. $R_a(s', s)$ can be thought of as the reward (signal) that the agent receives after having taken action $a$ in state $s$ and having landed in state $s'$.

In other words, the reward system is just this function $R_a(s', s)$, which is often the hardest function to define when modelling an RL problem. However, in certain cases, this function can be easily defined. For example, in the case of chess, you could define $R_a(s', s) = 1$, if $s'$ is the terminal state (checkmate), else $R_a(s', s) = 0$. Nonetheless, we could define the reward function for a chess environment differently. For example, we could give a reward of $0.5$ for certain "clever" moves. So, the reward function needs to be defined (by the programmer) in order to solve an RL problem and it highly affects the way the agent will learn.

The reward function can also be denoted by $R_a(s)$, which can be thought of as the reward that the agent receives after having taken action $a$ in state $s$ (no matter which state it lands in), or $R(s)$, which can be thought of as the reward the agent receives either when it enters or exits state $s$.

$\endgroup$
  • 1
    $\begingroup$ Worth adding that in general the output of all the reward functions you show is a distribution of scalar reward values. Although very common to just define it as a fixed value, it may actually be random in some environments, e.g. amount of money made on a bet. $\endgroup$ – Neil Slater May 11 at 8:49
  • $\begingroup$ @NeilSlater Thanks for pointing this out. I will edit my answer later to add this. $\endgroup$ – nbro May 11 at 10:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.