5
$\begingroup$

I am currently trying to learn reinforcement learning and I started with the basic gridworld application. I tried Q-learning with the following parameters:

  • Learning rate = 0.1
  • Discount factor = 0.95
  • Exploration rate = 0.1
  • Default reward = 0
  • The final reward (for reaching the trophy) = 1

After 500 episodes I got the following results:

enter image description here

How would I compute the optimal state-action value, for example, for state 2, where the agent is standing, and action south?

My intuition was to use the following update rule of the $q$ function:

$$Q[s, a] = Q[s, a] + \alpha (r + \gamma \max_{a'}Q[s', a'] — Q[s, a])$$

But I am not sure of it. The math doesn't add up for me (when using the update rule).

I am also wondering either I should use the backup diagram to find the optimal state-action q value by propagating the reward (gained from reaching the trophy) to the state in question.

For reference, this is where I learned about the backup diagram.

Improve this question
$\endgroup$
5
  • $\begingroup$ by using the update rule in the question then you are implicitly using the backup diagram. the way to calculate the optimal q-values is using the update rule in the question. $\endgroup$
    – David
    Commented Feb 21, 2021 at 21:02
  • $\begingroup$ So basically if I want to compute the optimal state 2 action South (where the agent is standing) considering the old q value =0.9 (q value of the red arrow where the agent is standing) and the max q(s',a') is the q value of the red arrow in the state beneath it ? $\endgroup$
    – Rim Sleimi
    Commented Feb 21, 2021 at 21:07
  • 2
    $\begingroup$ the value of taking south from the agents current location is equal to the immediate reward it receives + the (discounted) q-value for the state it transitions into and action it takes under the current policy. as you're interested in the optimal policy then you want the action to be the one that maximises the q-value so yes it would be $r + \gamma Q(s', 'south')$ where $s'$ is the state directly below the agents current state. $\endgroup$
    – David
    Commented Feb 21, 2021 at 21:10
  • 1
    $\begingroup$ remember though that if you would like to calculate the values 'from scratch' then using the Q-learning update rule is how you would go about it. The way I described is just the definition of a q-value. $\endgroup$
    – David
    Commented Feb 21, 2021 at 21:18
  • 1
    $\begingroup$ Yeah I realized that there is a difference between the update rule and the definition of Q value. Thanks again! $\endgroup$
    – Rim Sleimi
    Commented Feb 21, 2021 at 22:25

1 Answer 1

Reset to default
4
$\begingroup$

It seems that you are getting confused between the definition of a Q-value and the update rule used to obtain these Q-values.

Remember that to simply obtain an optimal Q-value for a given state-action pair we can evaluate

$$Q(s, a) = r + \gamma \max_{a'} Q(s', a)\;;$$

where $s'$ is the state we transitioned into (note that this only holds when obtaining the optimal Q-value, if we were using a stochastic policy then we would have to introduce expectations).

Now, this assumes that we have been given/obtained the optimal Q-values. To obtain them, we have to use the update rule (or any other learning algorithm) that you mentioned in your question.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .