You don't have "9 possible states" in your problem, you have those 9 board squares multiplied by all possible values that the numbers within them can take, in all combinations. If those are discrete integers within bounds, then your space is finite - as an example if each square could contain the number $1$,$2$ or $3$ then your state space would have size $3^9 \times 9 = 177147$ (the extra $\times 9$ is because you also need to track which square the agent is in, and that can occur in any combination with the square values). If instead the ranges are unbound, or numbers real-valued, then your state space is infinite.
It is still possible to solve RL problems with infinite state space. As your action space is simple and discrete, you can still use value-based methods.
The step you need to make is to look into approximation schemes. Sutton & Barto has a useful section on the simpler approximation schemes that you can use with linear approximators (or even the most basic state aggregation might work for you). In second edition that is chapters 9 and 10.
.
A very common approach with your kind of problem is to use a neural network and DQN model (Q-learning adapted to work with a neural network). If you search for tutorials on working with DQN, you should find hundreds, so you can filter down to something relevant and useful to you.
The very basics of DQN are:
- It's Q-learning, but instead of the action value table, you will train a neural network to estimate $Q(s,a)$
- Combining neural networks with off-policy RL can be unstable, so there are a couple of important changes to learning process:
- Instead of immediately updating from the most recent experience, all experience is buffered in a large table and random sampled mini-batches are taken from that table to train with.
- Instead of using the current estimator to bootstrap from, an old copy of it is used and updated occasionally (e.g. once every 1000 steps)
