I have a set of users that can be one of 3 types.
They will randomly request a service from the UAV which is a drone used as a Base Station.
The UAV (the agent) is tasked with allocating subchannels (a resource type) and power values (another resource type) to the different users, all while choosing a new position at every step to maximize the power values (I use a mathematical equation in my code to calculate that).
A penalty is returned in case a user didn't request a service but was allocated a resource, and a reward is returned otherwise.
The penalty's value varies depending on the number of the users that were allocated resources they didn't ask for.
The state is defined as a combination of a matrix of 0s and 1s, and the agent's (UAV) position (The agent is placed in a 2D grid).
The rows of the matrix represent the users, and the columns represent the type of users (3 types).
row == (0, 0, 0), it means that the user at that row is not requesting any services.
row == (1, 0, 0) or
row == (0, 1, 0) or
row == (0, 0, 1), it means that the user is requesting a service from the UAV. (There can be at most one column with the value of 1 in each row, we cannot have, for example,
row == (1, 1, 0) or
(1, 1, 1) etc)
The action space is defined as a combination of:
- Another matrix of 0s (or rather the letter 'Z') and 1s (or rather the letter 'O'), with the rows representing the first type of resources (subchannels) that the UAV has, and the columns represent the users.
- The agent's chosen movement (Up, Down, Left or Right).
- And a vector with the different power values that were assigned to the users, hence the vector's length = number of users.
As for the matrix, suppose we have 5 users and 10 subchannels, if subchannel 0 was allocated to User0, then the first row (
row) of the matrix would look like this:
(O, Z, Z, Z, Z)
(Each subchannel can be allocated to one user only, and each user can be allocated one subchannel only as well at every time step / So we have one 'O' only in each row and each column)
I have created a functioning Q-Learning Agent, taking a very small number of users, subchannels, power values and grid dimensions.
Grid rows = 5 Grid columns = 5 number of subchannels = 4 number of users = 3
Then, regarding the Q-table, we have
Q[state, action], with state representing each row of the state space, and the action is each possible action that can be taken when we have that state.
The state being every possible combination of:
- All the possible positions the UAV can take.
- And all the possible state matrices that we can have.
The action being every possible combination of:
- All the possible movement action chosen by the UAV.
- All the possible subchannel allocation matrices we can have.
- And all the possible power vectors with the values the UAV chose to give to the users.
So the Q-table is very huge already even with a very small sample.
For example, suppose we have:
state = [ 5, ( (0, 1, 0), (1, 0, 0), (0, 0, 0) ) ]
5 being the agent's position, and the 2D tuple is the matrix of requests. User0 and User1 are asking for a service, but User2 isn't.
Then one entry of the Q-table would be:
Q[state, (U, ((O, Z, Z), (Z, Z, Z), (Z, O, Z), (Z, Z, Z)), (12, 12, 0))] = value
The second tuple is the matrix that represents the allocation of the subchannels. We have 3 users, which means we have 3 columns.
We have 4 subchannels, which means we have 4 rows.
Z = means allocate zero to the user at that column, which means that the subchannel at that row was not allocated to the user at that column.
O = means that the subchannel at that row was allocated to the user at that column. (so the value of that action is 1)
U = the agent chose to go up so the value is (-1)
The third tuple is the power vector, it's the power values allocated to the users.
In this example, no penalty would be returned, because User0 and User1 who were allocated subchannels and non-zero power values, did actually ask for a service as we see in the requests matrix.
Obviously, I cannot scale up my sample because I get MemoryError, so I have to use a DQN agent. However, I'm having a hard time figuring out how to build my model architecture (Neural Networks and the like).
I hope someone can guide me.