1
$\begingroup$

Does it make sense to incorporate constant states in the Markov Decision Process and employ a reinforcement learning algorithm to solve it?

For instance, for applications of personalization, I would like to include users' attributes, like age and gender, as states, along with other changing states. Does it make sense to do so? I have a sequential problem, so I assume the contextual bandit does not fit here.

$\endgroup$
0
0
$\begingroup$

It is, I suppose, a philosophical question whether data that describes a whole episode and does not respond to events within it is part of the state, or is part of some other structure.

However, the practical response is to view such descriptive data as defining an instance of a class of related environments, and to include it in the state features. This may be done for two main reasons:

  • The static data is a relevant parameter of the environment, affecting state transitions and rewards.

  • It is possible to generalise over the population of all values that the parameters can take.

In simple environments, generalisation might only be that the same agent can learn about all variations in a single combined training session. You could use a tabular RL method, starting randomly with one of the possible variations until all were sufficiently covered.

In more complex environments, generalisation may also occur through functon approximation, in a similar manner to contextual bandits. In your personalisation example, you are not expecting to train the agent for all possible user descriptions, but hope that people with similar age, gender etc descriptions will respond similarly to an agent that personalises content.

Philosophically, the contextual data is either part of a larger state space (with a restriction that transitions between different contexts do not happen within an episode), or it is metadata that impacts the "real" state transitions and rewards. Pragmatically, to allow the data to influence value functions and policies, it is necessary to use it in the arguments of those functions. Whether you then view it as part of the state feature vector or as something that is concatenated to state features is a personal choice. Most of the literature I have seen assumes without comment that it is part of the state.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.