# How should I generate datasets for a SARSA agent when the environment is not simple?

I am currently working on my master's thesis and going to apply Deep-SARSA as my DRL algorithm. The problem is that there is no datasets available and I guess that I should generate them somehow. Datasets generation seems a common feature in this specific subject as stated in [1]

When a dataset is not available, learning is performed through experience.

I am wondering how to generate datasets when the environment is not as simple as a tic-tac-toe or a maze problem and what the experience means.

PS: The environment consists of 15 mobile users and 3 edge servers, each of which covers a number of mobile users. Each mobile user might generate a computationally heavy-load task and at the beginning of each timestep and can process the task itself or requests its associated edge server to do the processing. If the associated edge server is not capable of processing, due to some reasons, it requests a nearby edge server to lend it a hand. The optimization problem (reward) is to reduce time and energy consumption (multi-objective optimization). Each server has a DRL agent that makes offloading decisions.

I'd really appreciate your suggestions and help.

I am wondering how to generate datasets when the environment is not as simple as a tic-tac-toe or a maze problem

There is no difference in concept, which is why tic-tac-toe and maze problems are used to teach.

As you have noted, the main difference between reinforcement learning (RL) and supervised learning is that RL does not use labeled datasets. If you are using SARSA then you would not expect to use any record of previous experience either because SARSA is designed to work on-policy and online - which means that data needs to be generated during training. Training data for SARSA is typically stored only temporarily before being used, or is used immediately (you might keep a log of it for analysis or to help document your thesis, but that log will not be used for further training by the agent). This is different to Q-learning and DQN, which could in theory make use of longer-term stored experience.

You have two main choices for acquiring data:

• Use a real environment. In your case, set up 15 mobile users and 3 edge servers. Instrument the environment to collect state and reward data for the agent. Implement the agent as the real decision maker in this environment.

• Simulate the environment. Write a simulation that models user behaviour and server loading. Instrument that to provide state and reward data, and integrate your learning agent with it. Typically the agent will call the environment's step function, passing the action choice as an argument and receiving reward and state data back.

If you can simulate the environment, this is likely to be preferable to you since you will likely use less compute resources (than 3 servers and 15 mobile phones) and can run the training faster than real time. Deep reinforcement learning can use a large amount of experience to converge on near-optimal policies, and fast simulations can help because they generate experience faster than reality.

You can also do both approaches. Train an initial agent in simulation, then implement the real version once it reaches a good level of performance in simulation. You can even have the agent continue to learn and refine behaviour in production. Given that you are working with SARSA, this may be an important part of the intent of your project, that the agent continues to adapt to changes in user behaviour and server load over time. In fact this is a key advantage of SARSA over Q-learning, that it should be more reliable and safe to use in such a continuous learning scenario deployed to production.

and what the experience means.

The experience in reinforcement learning is the record of states, actions and rewards that the agent encounters during training.

• Thanks a lot for your clear explanation! – Shahin Jan 6 at 10:11
• It's not fully clear to me this part "You have two main choices for collecting data ...": so are you suggesting to "collect data" in the sense that the OP should really store a dataset first and then replay that dataset to perform SARSA somehow offline? Or are you suggesting to "collect data", but you're actually referring to where SARSA can be trained (i.e. gain experience)? It seems more the second, but the suggestion would be applicable to both cases, anyway. – nbro Jan 8 at 0:38
• @nbro: "Collecting data" is not meant to imply a dataset or offline training. I think I make clear the lack of labelling and duration of useful storage in the previous paragraph. However, there is still data involved in the proceess and it has to be acquired somehow. "Collect" also has the meaning of "fetch" (as in "I collected the kids from school"). I have changed to more neutral "acquire", but open to suggestions – Neil Slater Jan 8 at 7:39