1
$\begingroup$

I would like some guidance on how to design an Environment for a Reinforcement Learning agent where the stopping conditions and rewards for the environment change based on an initial set of input parameters.

For example, let's say that a system generated alert triggers the instantiation of the RL environment, whereby the RL agent is launched to make decisions in the environment, based on the alert. The alert has two priorities "HIGH" and "LOW", when the priority is "HIGH" the stopping condition is a reward of "100" and when the priority is "LOW", the stopping condition is a reward of "1000".

In this scenario, is it preferable to create two separate environments based on the priority (input parameter) of the alert? Or is this a common requirement that should be designed into the environment/agent? If so, how? Note that I have simplified the scenario, so there could be multiple conditions (e.g., alert, system type, etc), but I am just trying to find a basic solution for the general case.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

In this scenario, is it preferable to create two separate environments based on the priority (input parameter) of the alert?

It is difficult to make a hard rule here.

If the resulting environments can be cleanly sorted into a few different categories, and the ideal behaviour and/or the states visited are radically different in each scenario, then maybe a few different agents optimised for each scenario could work well.

A more general approach however, is to include the episode start data as part of the state that the agent observes on each time step. A single agent can then in theory learn the different behaviours required depending on the initial values, plus still generalise from anything shared between the multiple scenarios.

The alert has two priorities "HIGH" and "LOW", when the priority is "HIGH" the stopping condition is a reward of "100" and when the priority is "LOW", the stopping condition is a reward of "1000".

This may work against you. RL agents do not respond to the absolute values of rewards, other than how they compare to other rewards also available within the same episode (or continuing environment).

If there is only ever one issue to solve at a time, and no conflict between solving either of the "HIGH" or "LOW" priority problems (such as splittig resources or effort between them), the different reward system seems redundant. Solved is solved. You might rate the usefulness of an agent that solves the "LOW" priority issue well higher, but it seems to me that this describes what you should work on first, not the goals of the agent. To influence the goals of the agent, both rewards would need to be available within the same episode or continuing environment, requiring the agent to make a choice between them.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thanks @neil-slater, I should have clarified the Alert can only be one priority in the example I gave e.g., "HIGH" or "LOW", such that the RL agent can learn what is the best policy (and therefore series of steps to be taken) based on this priority. I feel myself leaning towards the first example you gave, but of course if one increases the number of parameters in the inputs e.g., Alert Priority, System Criticality, Alert Triage Time (where there is a maximum time that the episode can run for), then, the separate environments for the different combinations likely wouldn't scale well. $\endgroup$
    – RL_NOOB
    Nov 24, 2020 at 17:58
  • $\begingroup$ with this approach "A more general approach however, is to include the episode start data as part of the state that the agent observes on each time step. A single agent can then in theory learn the different behaviours required depending on the initial values, plus still generalise from anything shared between the multiple scenarios." <- conceptually how might one go about implementing this? Are there any toy examples you can point me in the direction of? $\endgroup$
    – RL_NOOB
    Nov 24, 2020 at 19:41
  • $\begingroup$ @RL_NOOB: Add extra features to describe the nature of the goal (perhaps a boolean, or a one-hot-encoded class), and train agents as normal. Make sure to train with all possible variations of the goal. If you want the agent to generalise then you will need to use some kind of approximator - e.g. neural networks are popular in agents like DQN. Using an approximator will generalise behaviour to work in previously unseen states (caveat: as well as e.g. neural networks usually do). I assumed you were using something like DQN in the answer but if your state is simple enough you might not need to $\endgroup$
    – Neil Slater
    Nov 24, 2020 at 19:52
  • $\begingroup$ @RL_NOOB: A clasic toy example where goal state is moved around is Taxi = gym.openai.com/envs/Taxi-v3 $\endgroup$
    – Neil Slater
    Nov 24, 2020 at 19:53
  • $\begingroup$ Thanks @neil-slater your comments and insight have been really helpful :) $\endgroup$
    – RL_NOOB
    Nov 24, 2020 at 20:36

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .