-1
$\begingroup$

I read this article about Q-learning: https://www.learndatasci.com/tutorials/reinforcement-q-learning-scratch-python-openai-gym/

It teaches how to implement the algorithm using the Gym Python library.

But, what are the steps of Q-learning from scratch?

Please, could someone can explain what are the basic steps of the Q-learning algorithm?

$\endgroup$
4
  • $\begingroup$ Hello @will The J and welcome to AI Stack Exchange. This question as posed is very broad. If possible, it may help to edit the question to ask a specific question about Q-learning. Even if you need to make multiple posts, you are much more likely to get answers by making a few specific questions than one broad question here. Thank you for participating, and we hope to see more of your questions here in the near future! $\endgroup$
    – DeepQZero
    Nov 4, 2023 at 15:44
  • $\begingroup$ Thank you very much for wanting to help! Okay, I edited the question to make it more specific. $\endgroup$
    – will The J
    Nov 4, 2023 at 16:03
  • $\begingroup$ Any book on RL should contain a description of Q-learning, which is like the most famous RL algorithm ever. Having said that I've written 1 or 2 answers that attempt to explain Q-learning, but I don't remember their title. You can search on the site. $\endgroup$
    – nbro
    Nov 10, 2023 at 1:55
  • $\begingroup$ Thanks for the sugestion $\endgroup$
    – will The J
    Nov 10, 2023 at 18:39

1 Answer 1

3
$\begingroup$

This is taken from Sutton & Barto's RL book

Algorithm parameters: step size alpha in (0, 1], small epsilon > 0
Initialize Q(s, a), for all s in S+, action a in A(s), arbitrarily except that Q(terminal, ·)=0
Loop for each episode:
Initialize S
Loop for each step of episode:
Choose A from S using policy derived from Q (e.g., epsilon-greedy)
Take action A, observe R, S'
Q(S, A) = Q(S, A) + alpha[R +  gamma * max(a) Q(S', a)  Q(S, A)]
S = S'
until S is terminal

What this means is

  1. Initialize you alpha between 0 and 1, and an epsilon > 0
  2. Initialize your Q for all states and actions i.e. if you have a 3x3 world with 4 actions (up, down, left, right) it would be a 3x3x4 array.
  3. Pick a number of episodes for the robot to move in the world to train for.
  4. Initialize your state (perhaps 0x0)
  5. Do these things for each step in the episode
  6. Choose an action a based on your state s in Q BUT allowing for randomness (exploration) based on your epsilon. Meaning, pick a random number between 0 and 1 and if it is greater than epsilon act randomly, otherwise pick the best/max action from Q.
  7. Feed the action to the world/environment and record the R/Reward S'/S Prime/next State.
  8. Update your Q array at Q(S, A) (i.e. Q[0, 0, 1] which would mean we are at 0, 0 and going up) with Q(S, A) (yes, that is the value of where you are) + alpha (usually a small value like .005) * (R/Reward + gamma * (max_aQ(S', A)(the best/max action a of S'/the next state) - Q(S, A)/the current state)

8 is clearer as a formula if you look at the formula in the algorithm. The point is to slowly increment Q(S, A) using alpha to the true reward value by tying it the the next State via a specific action. When we get to that true value we use our Q array (called exploitation) at any given state and pick the max value from the set of possible actions.

  1. Update S to the next state until we reach the terminal state (3x3 in my example)
$\endgroup$
1
  • $\begingroup$ Thanks for explaining $\endgroup$
    – will The J
    Nov 9, 2023 at 22:29

Not the answer you're looking for? Browse other questions tagged .