# DQN layers when state space and action space are multi dimensional

I have built my own RL environment, where a state is composed of two elements: the agent's position and a matrix of 0s and 1s (1 if a user has requested a service from the agent, 0 otherwise); an action is composed of 3 elements: the movement the agent chooses (up, down, left or right), a matrix of 0s and 1s (1 if a resource has been allocated to a user, 0 otherwise), and a vector representing the allocation of another type of resource (the vector contains the values allocated to the users).

I am currently trying to build a Deep Q Learning agent, I am a bit confused however as to what model (example Sequential), what type of layers (example Dense layers), how many layers, what activation mode I should use, and what the state and action sizes are. (Taking this code as a reference cartpole dqn agent)

I also do not know what my inputs and outputs should be.

The examples I have come across are rather simple and I don't know how to approach setting it all up for my agent.

• assuming your agents position is a vector in $\mathbb{R}^d$, you could have a network that takes as input the position and passes it through a fully connected layer, the matrix passed through a convolutional layer, and the concatenate the two outputs of those layers to pass through further fully connected layers. Nov 12 '20 at 21:14
Assuming your agents position is a vector in $$\mathbb{R}^d$$, then I would have the network take as input this position vector and pass it through a fully connected layer. I would also take as input the matrix and pass it through a convolutional layer(s) and flatten the output so it is now also a vector in $$\mathbb{R}^{d'}$$. I would then concatenate these together so you have a vector in $$\mathbb{R}^{d + d'}$$ and pass this through some fully connected layers as usual.
Your output would typically be a score for each of the action combinations, though as one of your outputs is a matrix this could potentially make things expensive to compute as you would need $$2^{n \times m}$$ binary representations just for the matrix ($$n$$ is the number of rows and $$m$$ is the number of columns).