1
$\begingroup$

I have created my own RL environment where I have a 2-dimensional matrix as a state space, the rows represent the users that are asking for a service, and 3 columns representing 3 types of users; so if a user U0 is of type 1 is asking for a service, then the first row would be (0, 1, 0) (first column is type 0, second is type 1...).

The state space values are randomly generated each episode.

I also have an action space, representing which resources were allocated to which users. The action space is a 2-dimensional matrix, the rows being the resources that the agent has, and the columns represent the users. So, suppose we have 5 users and 6 resources, if user 1 was allocated resource 2, then the 3rd line would be like this: ('Z': a value zero was chosen, 'O': a value one was chosen) (Z, O, Z, Z, Z)

The possible actions are a list of tuples, the length of the list is equal to the number of users + 1, and the length of each tuple is equal to the number of users. Each tuple has one column set to 'O', and the rest to 'Z'. (Each resource can be allocated to one user only). So the number of the tuples that have one column = 'O', is equal to the number of users, and then there is one tuple that has all columns set to 'Z', which means that the resource was not allocated to any users.

Now, when the agent chooses the action, for the first resource it picks an action from the full list of possible action, then for the second resource, the action previously chosen is removed from the possible actions, so it chooses from the actions left, and so on and so forth; and that's because each user can be allocated one resource only. The action tuple with all 'Z' can always be chosen.

When the agent allocates a resource to a user that didn't request a service, a penalty is given (varies with the number of users that didn't ask for a service but were allocated a resource), otherwise, a reward is given (also varies depending on the number of users that were satisfied).

The problem is, the agent always tends to pick the same actions, and those actions are the tuple with all 'Z' for all the users. I tried to play with the q_values initial values; q_values is a dictionary with 2 keys: 1st key: the state being a tuple representing each possible state from the state space, meaning (0, 0, 0) & (1, 0, 0) & (0, 1, 0) & (0, 0, 1), combined with each action from the possible actions list. I also tried different learning_rate values, different penalties and rewards etc. But it always does the same thing.

enter image description here enter image description here

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

I am confused. For the initial $Q$-values, you generate one for each possible row $(1, 0, 0), (0,0,0), \ldots$ so you would have 4 states.

However, from the first paragraph it seems that the states themselves are matrices (one row for each user), so the state space is a set of such matrices.

That means that your $Q$-table should have a row for each possible matrix, and a column for each possible total assignment of items to users.

Improve this answer
$\endgroup$
3
  • $\begingroup$ So, suppose we have Q-table accessed as follows: Q[state, action]. So the state is the whole state space matrix and the action is the whole matrix with the different actions for each resource and user ? $\endgroup$
    – Ness
    Commented Oct 21, 2020 at 16:17
  • $\begingroup$ I am confused too because from what I gathered from the examples online, is that if we have a state space that's an array of different positions (for example) that the agent can be in, then the state in the Q-table would be each position in that state space. So what I thought I should do is represent each row in the state space as a key for the Q-table. $\endgroup$
    – Ness
    Commented Oct 21, 2020 at 16:20
  • $\begingroup$ A state should be the full description of the state. The state space is the set of all possible such states. It appears that in your example, a state is a matrix (with one row for each current user). The state space is a set of such matrices. In the online examples, each state is just a single position, so there the table is a bit easier. $\endgroup$
    – Robby Goetschalckx
    Commented Oct 21, 2020 at 16:25

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .