To update the Q table Q-learning takes the arg max of the Q values - the state, value mappings.
For example, in tic tac toe the state
XOX OXO -X- contains two available positions, each marked by the
- character. In order to evaluate the arg max should
temporal difference calculate the arg max of just the available positions?
For the state
XOX OXO -X- the arg max should be taken at index 6 and 8 ? (assuming zero indexing) If not then how should the arg max indexes be updated? The indexes
0,1,2 3,4,5 7 can be used as they are already been taken by X or O so should not be evaluated for their value ? This also means that indexes which are not available will not have their Q value updated, will this break the Q-learning procedure ?