Many Q-learning techniques have been developed to capture discrete state(observation), actions like a robot in a grid world, and even continuous (state or action) spaces. But I am wondering how we can model the states/space in a time-dependent environment. Please, consider the following example:

There is one smartphone (client) and five compute servers that are addressing/serving many clients (smartphones) at the same time. The smartphone transfers some raw data (e.g, sensor data) to one of those five servers (e.g., every t seconds) and gets the results. Suppose the server computes the stress-level of the client in real-time based on the collected data.

Now, a q-learning agent should be deployed to the smartphone to be able to select the best server with minimum response time (i.e., the goal is to minimize the execution/response time). Note that servers are serving different clients and their load is a function of time and varies from time to time.

So in the above scenario, I am wondering what would be our "states" and how we can model the "environment"?

  • $\begingroup$ Does the client that needs to make a decision know the current load on the server, or have state data that would allow it to approximate it? Is it allowed to have state such as response statistics for its last few requests, or some statistical data from the servers for past equivalent time periods? $\endgroup$ – Neil Slater Apr 29 '20 at 12:30
  • $\begingroup$ Architecturally, in the real world, it is more normal to have server allocation managed inside a cluster by a load balancer server rather than by clients, since the service will be able to measure and compare load/responses across multiple services whilst the clients cannot do this. If you are trying to solve a real world problem to create a stabel and fast service, you should probably consider this approach first as opposed to AI at the client, but the AI issue is likely interesting and on-topic here regardless of that. $\endgroup$ – Neil Slater Apr 29 '20 at 12:33
  • $\begingroup$ Well, the smartphone can log and store its own last few requests. $\endgroup$ – user2867237 Apr 29 '20 at 12:44
  • $\begingroup$ Regarding load balancer: yes that is correct. I just simplified the problem. I am working on an offloading technique for IoT, in which the and edge device is connected to a fog node, and the fog node is connected to the cloud (with several servers). Now the edge should decide to run the task itself, or offload the task to the fog nodes, or to the cloud. $\endgroup$ – user2867237 Apr 29 '20 at 12:46
  • $\begingroup$ According to the system model, three different sites are considered as a feasible platform for offloading computation (1) the mobile fog in close proximity of end-user devices, i.e. site L1 (2) the adjacent mobile Fog (or distant mobile Fog) to handle mobility and load balancing issues, i.e. site L2 (3) the remote public cloud to manage huge traffic and computing requirements and archiving, i.e. site L3. So the actions are: 1) run in the device,, offload to L1, offload to L2, or offload to L3. $\endgroup$ – user2867237 Apr 29 '20 at 12:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.