I am trying to work on a variation of the Access-Control Queuing Task problem presented in Chapter 10 of Sutton’s reinforcement learning book .
Specific details of my setup are as follows:
- I have different types of tasks that arrive to a system (heavy/moderate/light with heavy tasks requiring more time to be processed.). The specific task type is chosen uniformly at random. The task inter-arrival time is $0.1s$ on average.
- I have different classes of servers that can process these tasks (low-capacity; medium capacity; high capacity; with high capacity servers having a faster processing time). When I select a specific server from a given class, it becomes unavailable during the processing time of the task assigned to it. Note that the set of servers (and as a result the number of servers of each class) is not fixed, it instead changes periodically, according to the dataset used to model the set of servers (so specific servers may disappear and new ones may appear, as opposed to the unavailability caused by the assignment). The maximum number of servers of each class is $10$.
- My goal is to decide which class of server should process a given task, in a way that minimizes the sum of the processing times over all tasks.
The specific reinforcement learning formulation is as follows:
- State: the type of task (heavy/moderate/light) ; the number of available low capacity servers; the number of available medium capacity servers; the number of available high capacity servers
- Actions: (1) Assign the task to a low capacity server (2) assign the task to a medium capacity server (3) assign the task to a high capacity server (4) a dummy action that has a worse processing time than the servers with low capacity. It is selected when there are no free servers.
- Rewards: the opposite of the processing time, where the processing times are as follows (in seconds):
| | Slow server | Medium Server | Fast server | "Dummy action" | |---------------|-------------|---------------|-------------|----------------| | Light task | 0.5 | 0.25 | 0.166 | 0.625 | | Moderate task | 1.5 | 0.75 | 0.5 | 1.875 | | Heavy task | 2.5 | 1.25 | 0.833 | 3.125 |
My intuition for formulating the problem as an RL problem is that 'Even though assigning Light tasks to High capacity servers (i.e. being greedy) might lead to a high reward in the short term, it may reduce the number of High capacity servers available when a Heavy task arrives. As a result, Heavy tasks will have to be processed by lower capacity servers which will reduce the accumulated rewards'.
However, when I implemented this (using a deep Q-network specifically), and compared it to the greedy policy, I found that both approaches obtain the same rewards. In fact, the deep Q-network ends up learning the greedy policy.
I am wondering why such a behaviour occured, especially that I expected the DQN approach to learn a better policy than the greedy one. Could this be related to my RL problem formulation? Or there is no need for RL to address this problem?
Sutton, R. S., & Barto, A. G. (1998). Introduction to reinforcement learning (Vol. 135). Cambridge: MIT press.
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., ... & Petersen, S. (2015). Human-level control through deep reinforcement learning. Nature, 518(7540), 529-533.f