# How do I convert table-based to neural network-based Q-learning?

I've used a table to represent the Q function, while an agent is being trained to catch the cheese without touching the walls.

The first and last row (and column) of the matrix are associated with the walls. I placed in last cell a cheese that agent must catch while being training.

So far, I've done it with dynamic states and, when necessary, I resized matrix with new states. I've used four actions (up, left, right and down).

I would like now to use an ANN to represent my Q function. How do I do that? What should be the input and output of such neural network?

A neural network (NN) is a "function approximator", that is, it is a model that can be used to approximate functions. In fact, a neural network with at least one hidden layer is a "universal" function approximator (that is, it can approximate any function).

In mathematics, a function $$f$$ is usually represented as a mapping of the form $$f: D \rightarrow C$$, where $$D$$ and $$C$$ are respectively the domain (inputs) and codomain (outputs) of $$f$$, and $$\rightarrow$$ means that $$f$$ "maps" $$D$$ to $$C$$. A NN (with at least one hidden layer) can thus approximate any function $$f$$ of this form.

In your context, the $$Q$$ table is a function: it is a mapping between states and actions (inputs) and Q values (outputs), which are the "expected future cumulative reward" (that you will obtain if you take a certain action from a certain state and then continue to follow the same policy). The Q function can thus more formally be denoted by $$Q: (S, A) \rightarrow \mathcal{R}$$, where $$S$$ is the "space of states" and $$A$$ is the "space of actions" in your problem. Initially, your Q table does not contain the correct (optimal) values. However, after training (or learning), hopefully, your Q table will be an approximation to the optimal Q function for your specific problem.

How do you then represent this table using a NN? What should be the inputs and outputs of the NN?

Let's suppose that your $$Q$$ table is implemented as matrix $$M$$. Then $$M[s, a]$$ is the $$Q$$ value for the state $$s$$ and action $$a$$. So, in this case, the combination of $$s$$ and $$a$$ is the input, whereas $$M[s, a]$$ is the output of your $$Q$$ function.

To represent this table as a NN, you can thus have as input of the NN the state and action, and as output the $$Q$$ value. You will then train the NN (using e.g. back-propagation) to learn the $$Q(s, a)$$, given a state $$s$$ and an action $$a$$ as input, for all $$s \in \mathcal{S}$$ and $$a \in \mathcal{A}$$. So, during your $$Q$$-learning algorithm, instead of using $$M[s, a]$$ to represent $$Q(s, a)$$, you will simply use the current output of your neural network.

Note that, in practice, I think, you will likely encounter problems while training the NN, if you update the weights of your NN at every time step (because e.g. NNs do not cope well with correlated data, and, in general, the "experience" data you will obtain from time step to time step will be highly correlated). Anyway, this is the basic idea of how to use a NN to represent a $$Q$$ function. There are other ways, but this is the simplest one, at least, conceptually.

• I've used 2D-array that represent Q-Matrix and all elements of array are iterating to form a string that's called state of Q-Matrix.So how can I set state as input of neuron while neuron input must be in float type and weight values are also used between -1 and 1 that is to update later on. – Arslan Ali Feb 9 at 16:37
• Why are you creating a string out of the 2d array? – nbro Feb 9 at 21:29
• So,How can I create states?It's represented matrix state where the agent exist recently and I'm adding it into list with resizing of matrix in case of dynamic states or newer states. – Arslan Ali Feb 10 at 8:20
• State of matrix can not be presented in floating points.It's possible only for distance based applications or other related to positions of agents in 2d-3d surfaces. – Arslan Ali Feb 10 at 8:22
• how to get state of matrix in float Number – Arslan Ali Feb 10 at 9:12