I'm having trouble understanding the 5th step in the flowchart.
For the 5th step, the 'update the Q function by taking the average of returns' is confusing.
From what I understand, the Q function is basically the state-action pair values put in a table (the Q table). To update it means to make adjustments to the state-action pair value of the individual states and their respective actions (e.g state 1 action 1, state 3 action 1, state 3 action 2, so on and so forth).
I'm not sure what 'average of returns' means though. Is it asking me to take the average of the returns after $x$ episodes? From my understanding, returns is the sum of rewards in a full episode (So, AVG=sum of returns for x episodes/x).
And what do I do with that 'average'?
I'm a little confused when they say 'update the Q function' because the Q function consists of many parameters that must be updated (the individual state-action pair value), and I'm not sure which one they are referring to.
What is the point of calculating the average of returns? Since the state-action pair value for a particular state and particular action will always be the same (e.g if I always take action 3 in state 4, I will always get value=2 forever)