How can certain information about the goal be given to the RL learning algorithm?

Question

Policy learning refers to mapping an agent state onto an action to maximize reward. A linear policy, such as the one used in the Augmented Random Search paper, refers to learning a linear mapping between state and reward.

When the entire state changes at each time-step, for example in the Continuous Mountain Car OpenAI Gym, the position and speed of the car changes at each time-step.

However, assume we also wanted to communicate the constant position of one or more goals. By "constant", I mean does not change within a training episode, but may change between episodes. For example, if there was a goal on the left and right of the Mountain Car.

Are there examples of how this constant/static information be communicated from the environment other than appending the location of the two goals to the state vector? Can static/constant state be differentiated from state which changes with each action?

Answer 1

I see three main ways to do this, which one makes more sense will depend on your application.

One is to append that information to the state/observations like you mentioned. While this information is static for a particular episode, it will be different across episodes and the policy should learn to condition the actions it chooses based on this information.

Another would be to leave goal information out entirely and force the agent to learn a policy that works when the goal is unknown. This will likely be more difficult to learn and you may end up with a policy that moves to the average of all goals or explores and tries them all.

A third option and probably the most natural is to have some context cue that the agent can observe (e.g. often in lab experiments with rats, a cue is placed on a wall that lets the rat know which way to go to get a reward). This is similar to the first method, except that the cue has to be observed rather than given directly. For the Mountain Car example, this could be an extra signal that the agent only sees in a particular location, such as the bottom of the valley or when it moves close to a particular side.