# What is correct update when the some indexes are not available?

To update the Q table Q-learning takes the arg max of the Q values - the state, value mappings.

For example, in tic tac toe the state XOX OXO -X- contains two available positions, each marked by the - character. In order to evaluate the arg max should temporal difference calculate the arg max of just the available positions?

For the state XOX OXO -X- the arg max should be taken at index 6 and 8 ? (assuming zero indexing) If not then how should the arg max indexes be updated? The indexes 0,1,2 3,4,5 7 can be used as they are already been taken by X or O so should not be evaluated for their value ? This also means that indexes which are not available will not have their Q value updated, will this break the Q-learning procedure ?

What you are referring to as the situation where

some indexes are not available

is simply the situation where some actions are not available/valid in some state. So, yes, the $${\arg \max }$$ will be calculated based only on the available actions in that state. More formally, $$\underset{a \in \mathcal{A}(s)}{\arg \max } \, Q(s, a)$$

where $$Q(s,a)$$ has been initialized for all $$s \in \mathcal{S}^{+}$$ and $$a \in \mathcal{A}(s)$$ and $$\mathcal{A}(s)$$ is defined as the set of all actions available in state $$s$$. See Sutton and Barto's Intro to RL book, chapter 6 (the part on Q-learning).

In the same book (chapter 3), the authors state that:

To simplify notation, we sometimes assume the special case in which the action set is the same in all states and write it simply as $$\mathcal{A}$$.

Therefore, in many sources, you may see the $$\arg \max$$ expressed as $$\underset{a \in \mathcal{A}} {\arg \max}$$ or even $$\underset{a} {\arg \max}$$. It's implicit that it is taken with respect to the set of actions $$\mathcal{A}(s)$$ available in state $$s$$.