# DQN layers when state space and action space are multi dimensional

I have built my own RL environment, where a state is composed of two elements: the agent's position and a matrix of 0s and 1s (1 if a user has requested a service from the agent, 0 otherwise); an action is composed of 3 elements: the movement the agent chooses (up, down, left or right), a matrix of 0s and 1s (1 if a resource has been allocated to a user, 0 otherwise), and a vector representing the allocation of another type of resource (the vector contains the values allocated to the users).

I am currently trying to build a Deep Q Learning agent, I am a bit confused however as to what model (example Sequential), what type of layers (example Dense layers), how many layers, what activation mode I should use, and what the state and action sizes are. (Taking this code as a reference cartpole dqn agent)

I also do not know what my inputs and outputs should be.

The examples I have come across are rather simple and I don't know how to approach setting it all up for my agent.

• assuming your agents position is a vector in $\mathbb{R}^d$, you could have a network that takes as input the position and passes it through a fully connected layer, the matrix passed through a convolutional layer, and the concatenate the two outputs of those layers to pass through further fully connected layers. Nov 12, 2020 at 21:14
• Genius ! Thank you so much for your answer ^^
– Ness
Nov 12, 2020 at 21:37
• I'll add my comment as an answer since you seem happy with it. :-) Nov 18, 2020 at 11:43

## 1 Answer

This is not necessarily the only way to do this but it would be the approach I'd take.

Assuming your agents position is a vector in $$\mathbb{R}^d$$, then I would have the network take as input this position vector and pass it through a fully connected layer. I would also take as input the matrix and pass it through a convolutional layer(s) and flatten the output so it is now also a vector in $$\mathbb{R}^{d'}$$. I would then concatenate these together so you have a vector in $$\mathbb{R}^{d + d'}$$ and pass this through some fully connected layers as usual.

As for how many layers and which activation to use this is something you'll have to do by trial and error as this really is problem specific.

Your output would typically be a score for each of the action combinations, though as one of your outputs is a matrix this could potentially make things expensive to compute as you would need $$2^{n \times m}$$ binary representations just for the matrix ($$n$$ is the number of rows and $$m$$ is the number of columns).