2
$\begingroup$

Typically, a Reinforcement Learning learning problem is formalized as finding an optimal policy for a Markov Decision Process (MDP). In many real-life situations, however, an agent can only get partial information from the environment. For example, Partially Observable MDPs are used to model the case where the agent does not fully observe the state.

I was wondering whether there is any well-established formalism for the case where the agent does not fully observe the reward signal.

In particular, I am thinking about the case where for every state-action pair $(s, a)$ the agent receives the reward $R(s, a)$ with probability $1 - \varepsilon$ and does not receive anything with probability $\varepsilon$. Of course, in principle, this setting can be thought as a regular MDP with a stochastic reward, but here I would like the agent to behave optimally w.r.t. to $R$.

I would really appreciate if you could point some relevant literature to me!

Improve this question
$\endgroup$
5
  • $\begingroup$ This does not directly answer your question (I think), but there are ways of dealing with problems where the reward function (RF) is not specified or it's not easy to specify it correctly. The basic idea is to learn the RF itself (this is sometimes known as inverse RL or reward learning, especially when you are given an optimal policy: that's why it's called IRL because, usually, in an RL problem, you want to find the policy from the RF), but you can also use behaviour cloning (i.e. you don't use a reward function at all, and learn a policy from examples taken from another policy). $\endgroup$
    – nbro
    Dec 1, 2021 at 10:13
  • $\begingroup$ @nbro Thanks for the comment! Good point! In fact I asked this question precisely because I am reading about IRL at the moment :) Oversimplifying, RL deals with MDP with observable reward function and IRL deals with the case where the reward function is missing. I was wondering whether some intermediate scenarios have been studied.. $\endgroup$
    – Onil90
    Dec 1, 2021 at 10:29
  • 1
    $\begingroup$ I am not sure if it's intermediate, but in the paper "Inverse Reward Design", if I remember correctly, they formulate the problem of learning a "correct" function from "observed RFs" (i.e. non-optimal or partial/incomplete ones) in a Bayesian/probabilistic setting. So, in a way, if I remember correctly, they kind of combine IRL with RL because they need to solve the RL problem too (but I could be recalling that wrongly, so you may want to check the paper). $\endgroup$
    – nbro
    Dec 1, 2021 at 11:00
  • $\begingroup$ Let me know if this answers your question so that I eventually give a formal answer below with this info. $\endgroup$
    – nbro
    Dec 1, 2021 at 11:02
  • $\begingroup$ @nbro Thanks a lot for the paper! I'll have a look and I'll let you know! $\endgroup$
    – Onil90
    Dec 1, 2021 at 15:01

1 Answer 1

Reset to default
2
$\begingroup$

Your setting (of randomly dropping out reward signals) impacts expected future reward by multiply everything by a common factor $(1-\epsilon)$.

As reinforcement learning (RL) control is based on maximising expected future reward, and multiplying by a positive constant does not affect ranking of action values, all existing RL methods will cope just fine without modification. They will behave optimally in the limit of training - with all the usual caveats of course - although learning will be slower due to added variance, and value estimates lower.

If the agent is allowed to observe that the reward signal is missing (as opposed to being an observed zero reward), then it could additionally estimate $\epsilon$, and correct its learned value function. I would recommend this is handled at the end of training as a separate function instead of modifying $Q(s,a)$ during training. That is because initial estimates for $\epsilon$ are likely to be inaccurate and make learning even slower (at least for TD learning due to bootstrapping on less accurate values).

Whether any corrected value function makes sense will depend on what it means to "not observe" the reward, and what the reward represents in terms of the environment and agent's goals. You may not need to know if all you care about is whether the agent is behaving optimally.

The setting becomes more complex if $\epsilon$ is made a function of $a$, $s$ or both, and solutions will be different depending on whether this function is known or unknown.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thank you very much! You are totally right.. Yeah, the way I have formulated the problem is actually very trivial, as you correctly noticed. As you pointed out, the situation is less trivial if $\varepsilon$ depends on $s$ and/or $a$. Do you think there could be any other meaningful notion of "partial observability" for the reward function? $\endgroup$
    – Onil90
    Nov 30, 2021 at 16:43

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .