3
$\begingroup$

I have a Reinforcement-Learning environment where the state is an array of 0s and 1s with length equals to the number of users the agent must satisfy (11 users).

The agent must choose one of 12 resources for the 11 users according to the state array. If state[0] == 1, that means that user0 needs a resource, so the agent must choose a resource out of the 12 resources it has. So, the action array's first element would be, for example: action[0] = 10, which means that resource 10 was allocated to user0.

If the next user (user1) is asking for a resource as well, then the number of resources to choose from is 12 - 1, in other words, because resource10 was already allocated to user0, it cannot be allocated to another user.

If state[X] == 0, it means that userX is not asking for a resource, therefore it must not be allocated any resource.

An example of a state array:

[1, 1, 0, 0, 0, 1, 1, 1, 0, 0, 0]

An example of an action array according to the state array example: (resource count starts at 0 | -1 indicates no resource was allocated)

[10, 2, -1, -1, -1, 3, 11, 5, -1, -1, -1]

I'm new to Reinforcement Learning and Deep Learning, and I have no idea how to translate that into a neural network.

Improve this question
$\endgroup$
2
$\begingroup$

Let's assume you would like to work with a classic DQN. You need to train the Q-network where inputs are the states and actions. The DQN is a function of Q(states, actions). The network is supposed to predict Q-value. The agent must pick up the action that produces the highest Q by giving the all possible actions to the network, in your case. Let's assume the current state is

[1, 1, 0, 0, 0, 1, 1, 1, 0, 0, 0]

You must pick up an item for the user which state is 1? you need to generate the all possible actions, for example,

[1, 2, -1, -1, -1, 3, 4, 5, -1, -1, -1]
[12, 1, -1, -1, -1, 2, 3, 4, -1, -1, -1]
[11, 12, -1, -1, -1, 1, 2, 3, -1, -1, -1]
...

You will need to generate all of these possible actions into the Q-network. Your worst case would be around 479 million possible actions to calculate in DQN.

If you would like to implement DQNs for this problem, I do recommend you to check out REINFORCE -> DQN -> Advantage Actor-Critic (A2C), which is the combination of policy network and DQN. With A2C, you can produce continuous actions with a policy network, action in range a[i]∈(0,1)*12, then probably round the generated actions. Then, the next problem is the policy network may produce an impossible action. So, you might skip to the nearest possible item.

For further reading on continuous action domains, I recommend to check out DDPG, PPO

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thank you so much :) $\endgroup$ – GeezLouiz Dec 7 '20 at 10:45
  • $\begingroup$ I'm struggling a bit with building the neural network itself, I get that I should use CNN, but I'm a bit confused as to what layers to use and how the input and output layers would look like. $\endgroup$ – GeezLouiz Dec 7 '20 at 13:38
  • $\begingroup$ If your inputs are not images, You don't need CNN. the main reason of using CNN is to extract the features, like in Atari paper where inputs are pixels of the game. I see your states and actions are raws, you probably use need to build FCC (Dense layers). here is a nice explanation. $\endgroup$ – Supawat Dec 7 '20 at 14:47
  • $\begingroup$ Thank you, I'll check it out :D $\endgroup$ – GeezLouiz Dec 7 '20 at 15:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.