7
The motivation for adding the discount factor $\gamma$ is generally, at least initially, based simply in "theoretical convenience". Ideally, we'd like to define the "objective" of an RL agent as maximizing the sum of all the rewards it gathers; its return, defined as:
$$\sum_{t = 0}^{\infty} R_t,$$
where $R_t$ denotes the immediate reward at time $t$. As ...
3
The value of a state depends on the policy that you use, so I'll make the assumption here that you're talking about value using the optimal policy.
According to the optimal policy, the agent would choose to stay in the square (1,1) every time, but since it has a 0.8 probability of actually staying (and 0.2 probability of dying), we can compute the value of ...
2
The discount factor is not something you should be optimising. It is typically part of the problem statement.
For practical purposes, you may set it below 1.0 for continuous problems when in fact you care about best long-term reward. Another option to avoid infinities on continuous problems is to re-formulate the problem as optimising average reward. A high ...
2
You know all the rewards. They're 5, 7, 7, 7, and 7s forever. The problem now boils down to essentially a geometric series computation.
$$
G_0 = R_0 + \gamma G_1
$$
$$
G_0 = 5 + \gamma\sum_{k=0}^\infty 7\gamma^k
$$
$$
G_0 = 5 + 7\gamma\sum_{k=0}^\infty\gamma^k
$$
$$
G_0 = 5 + \frac{7\gamma}{1-\gamma} = \frac{5 + 2\gamma}{1-\gamma}
$$
2
I will fill in some details in shaabhishek's answer for people who are interested.
With this in mind, what is the value of a square (1,1)?
First of all, the value function is dependent on a policy. The supposed correct answer you provided is the value of $(1, 1)$ under the optimal policy, so from now on, we will assume that we are finding the value ...
1
There are a few ways to resolve values of infinite sums. In this case, we can use a simple technique of self-reference to create a solvable equation.
I will show how to do it for the generic case here of an MDP with same reward $r$ on each timestep:
$$G_t = \sum_{k=0}^{\infty} \gamma^k r$$
We can "pop off" the first item:
$$G_t = r + \sum_{k=1}^{\...
1
Personally, I find the best way to think of SMDPs intuitively by just imagining that you just discretise the time into such small steps (infinitesimally small steps if necessary) that you can treat it as a normal MDP again, but with some extra domain knowledge that you can exploit primarily for computational efficiency:
Only at time steps that really ...
1
This "decay" of later values is a direct consequence of the episodic formula for the objective function for REINFORCE:
$$J(\theta) = v_{\pi_\theta}(s_0)$$
That is, the expected return from the first state of the episode. This is equation 13.4 in the book edition that you linked in the question.
In other words, if there is any discounting, we care less ...
1
First, an important note on any form of discounting: adding a discount factor can change what the optimal policy is. The optimal policy when a discount factor is present can be different from the optimal policy in the case where a discount factor is absent. This means that "artificially" adding a discount factor is harmful if we expect to be capable of ...
Only top voted, non community-wiki answers of a minimum length are eligible
