I have a problem statement where a couple of smartphones, for example inside a shopping mall, can migrate their time-consuming tasks/processes like image processing to an edge server located nearby. The system state space is represented as a $2n+1$ vector including task features like estimated execution time and energy consumption, as well as, the edge server's process queue where $n$ is the number of tasks at each timestep. The action space is represented as a $2n$ binary vector indicating to migrate each task or not. The goal is to reach a trade-off between execution time and energy consumption in the system.

Well, the problem is that the agent doesn't learn and I guess it might have to do with either state representation or action constraints that the environment induces: (an episodic environment)

  1. State representation at each timestep: I feed a $2n+1$ vector to the neural model. Whenever an action is taken on a task (for example, indices 8 and 9 in the action vector represent actions for task #4), execution time and energy consumption features for that task in the state vector become zero in order to teach the model that it should not consider taking an action on it again i.e., smaller Q-values.

  2. Action constraints: There are two types of invalid actions: I) If a decision is made for a task at the previous timestep, the actions corresponding to that task are considered invalid/illegal at the current timestep (I don't want the agent to be stuck in a loop, making decisions for the same task over and over again). II) If a task's running prerequisite is not already present in the edge server then that task cannot be migrated. For example, task #4 action is limited only to action #8 (action #9 is considered invalid/illegal) at the current timestep.

Deep reinforcement learning has been widely used in this field of study but I wonder if the number of invalid/illegal actions or the way I represent the system state is really keeping the agent from learning as to how to optimize the system.

I'd really appreciate any help and suggestions.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.