This question is similar to this question, however it has a different question.

I'm learning MDP's and I'd like to know if I'm doing these exercises correctly:

Consider the following MDP:

enter image description here

Suppose a behavioural scientist was doing an experiment where they gave rewards to an animal. Suppose $X=5$ (e.g., 5 food pellets at that location). It turns out that, after repeatedly exploring the grid, the animal seems to prefer going up to the rewards of $+2$.

Question 1: In this setting, prove that no scalar discount $\gamma\in[0,1]$ exists for which the optimal policy is to go to $+2$.

Question 2: Consider a Monte Carlo return $$G_t=R_{t+1}+f(R_{t+1}+f(R_{t+2}+f(...))).$$

Standard discounting can be seen as applying a linear transformation $f(x) = \gamma x$, by multiplying the remaining return after each step by a factor $\gamma$. Consider the following alternative where instead of multiplying with a factor $\gamma$, we raise the value to power: $f(x)=x^{\gamma}$. Does this mathematical model better explain the observed behaviour, in the sense that a $γ$ exists for which the optimal policy goes to $+2$? If so, give such a value for $γ$. If not, prove why not.

My answer to question 1: For the optimal policy to go to $2$, we need the return for going to $+2$ to be greater than both the return of going to $+1$ and $+5$, i.e., mathematically

\begin{align} 2\gamma > \gamma^25 \cap 2\gamma>1 \\ \frac{2}{5} > \gamma \cap \gamma > \frac{1}{2} \end{align}

Since $(\frac{1}{2},\infty) \cap (-\infty, \frac{2}{5}) = \emptyset$, this means that there is no such $\gamma$ for which the optimal policy is $+2$.

Answer to question 2:

Again the return of $+2$ should be both greater than $+1$ and $+5$:

\begin{align} 2^{\gamma} > (5^{\gamma})^{\gamma} \cap 2^{\gamma}>1 \\ \gamma \ln 2 > \gamma^2 \ln 5 \cap \gamma > 0 \\ \frac{\ln 2}{\ln 5} > \gamma \cap \gamma > 0 \\ 0.43 > \gamma \cap \gamma > 0 \end{align}

Hence for $0< \gamma < 0.43$, the optimal policy will be to go to $+2$. Is this correct?

  • 1
    $\begingroup$ Why are people voting to close this as opinion-based? So, now mathematics is an opinion?! What the...! $\endgroup$
    – nbro
    Jul 7, 2022 at 16:12
  • $\begingroup$ @nbro The 3 questions look very similar so I was confused thinking they are duplicates $\endgroup$ Jul 9, 2022 at 8:47
  • $\begingroup$ @ArayKarjauv Ha, ok, don't worry then! ;) $\endgroup$
    – nbro
    Jul 9, 2022 at 18:35
  • $\begingroup$ Both look correct $\endgroup$
    – David
    Aug 29, 2022 at 0:21


You must log in to answer this question.