I am watching David Silver's lectures on RL available on YouTube. My question here is with regard to Lecture 2 (Link to Video). At 1:11:00, I could not understand how he is calculating the state-value functions for C1, C2 and C3 (nodes with values 6, 8 and 10 respectively) in the student MDP example, starting from C3 and working backwards. Can someone please explain this?
$\begingroup$
$\endgroup$
3
-
$\begingroup$ I couldn't find any states labeled with C1 C2 or C3 at the given timestamp. $\endgroup$– tnfruCommented Sep 10, 2021 at 18:13
-
1$\begingroup$ Hello. It may also be a good idea to provide at least a diagram or a screenshot of it that illustrates the problem, if available. $\endgroup$– nbroCommented Sep 10, 2021 at 18:34
-
$\begingroup$ @tnfru Sorry about the lack of clarification! C1, C2 and C3 are the three nodes in the center of the diagram! (nodes marked with values 6, 8 and 10 respectively at the mentioned timestamp!) $\endgroup$– Gokulakrishnan CANDASSAMYCommented Sep 11, 2021 at 20:06
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
I looked at this not long ago. You need to understand that the slide is referring to an optimal (not expected) value function & optimal Action-Value function. Let's look at his diagram. From left to right you have state C1 = 6 for its optimal value, C2 has 8 and C3 has 10.
If we take C10, we can see there are 2 choices for the transition, 1) to the pub for a reward of +1 or 2) to study for +10 reward. There are no probabilities assigned to our decision, so we will take the action that maximizes our action-value. So, being in C3 and deterministically choosing to study gives a reward of 10. The action is studying and the reward is 10 so the action-value is 10 + the undiscounted value of the next state. From there you end up in the sleep state (with value 0) with no further reward.
If you are in C3 and deterministically choose the action to go to the pub, you receive a reward of +1 with 3 possibilities of states/values. Given this action value of C3 & going to pub, you can apply an expectation over values of where you end up (not in the choice you make from C3). Since you can end up in C1, C2 or C3 with their own probabilities and values, you end up with an action-value of 8.3 by choosing to go to the pub.
Likewise for calculating optimal action values for C1 and C2. You see which deterministic action gives the maximum sum of reward and next state.
