# DQN Agent with a 2D matrix as input in Keras

I have a Reinforcement Learning environment where the state is a 2D matrix with 0s and 1s (only one column with the value of 1 in each row).

Example:

(
(0, 1, 0),
(0, 0, 1),
(1, 0, 0),
(0, 0, 0),
(0, 1, 0)
)


The action the agent must take is for each row in the input, choose one resource out of 12 resources the agent has if there is a column with the value of 1 in that row, else choose no resource if the row has 0s only (example: row[3] wouldn't have any resources chosen for it by the agent). The rows correspond to the users the agent must allocate resources to.

In the step() method in the RL environment, the agent would receive a reward or a penalty depending on the action. If the reward is positive, the agent updates the state matrix, putting a 0 instead of 1 in the rows corresponding to the users that were allocated resources, which should be the next state. If the reward is negative, the episode ends, the environment resets and a new state is received by the agent

It came to my understanding that, in a deep learning approach, the DQN agent would receive a 2D matrix of 0s and 1s as input to its neural network (the state matrix), and output a vector with the chosen resources for each row of the input.

The network must choose a resource out of 12 resources for each row if that row has a 1 in it, and no resource is chosen if there is no column with the value of 1 in that row of the input. In other words, the network must choose an element out of 12 and output a vector with the chosen elements, depending on the input matrix.

Is there a way to do this using Deep Q-Learning and neural networks ?

• I am having trouble picturing this - could you give an example so that it is clearer what " only one column with the value of 1 in each row" means, and also what the action selection looks like? Am I right in thinking there must be more rows than columns in this problem (or at least as many)? Also you should explain the consequence of the choice - is this to optimise some measure of success/results from the action vector, and if so what happens next? – Neil Slater Dec 4 '20 at 18:51
• For Q-learning to apply, two things should happen - 1) there needs to be some measure of usefulness or success, and 2) the input (state) should repeat with a variation, and the possible variations depend somehow on the action vector. It is not at all clear what these things might be from your description – Neil Slater Dec 4 '20 at 18:55
• @NeilSlater thank you for your comment, I edited my post. I hope it's better explained. Answering your questions, yes there are more rows than columns. In the step() method in the environment, the action choice is rewarded according to the constraints I have defined. – Ness Dec 4 '20 at 19:11
• Thank you Ness, that is much clearer. I am still not sure what the purpose is though. Assuming for example that the agent chose (1, 5, 7, 0, 12) for allocating resources (where 0 is "no resource"), what would happen next? It is very important for choosing DQN and RL in general over some other algorithm: Will the values in the next matrix presented to the agent be in any way dependent on what the agent chose? Or another way to put things - are you trying to optimise a single function, or is there a time element with the goal of getting the best total result over multiple related inputs? – Neil Slater Dec 4 '20 at 19:21
• @NeilSlater originally, in my Q-learning agent, the action is another matrix of 0s and 1s, so row[0][1] = 1 would mean that resource0 was allocated to user1. But I got stuck trying to figure out how to do that. So, I had the idea of having the network choose from the resources. So, for example, the vector would have 10 elements (because we have 10 users (equals number of rows in the state matrix)), vector[1] = 5 would mean that resource5 was allocated to user1. That would help with the reward calculation later. I hope I answered your question. – Ness Dec 4 '20 at 19:27