# Policy and Discount Factor

This question is similar to this question, however it has a different question.

I'm learning MDP's and I'd like to know if I'm doing these exercises correctly:

Consider the following MDP: Suppose a behavioural scientist was doing an experiment where they gave rewards to an animal. Suppose $$X=5$$ (e.g., 5 food pellets at that location). It turns out that, after repeatedly exploring the grid, the animal seems to prefer going up to the rewards of $$+2$$.

Question 1: In this setting, prove that no scalar discount $$\gamma\in[0,1]$$ exists for which the optimal policy is to go to $$+2$$.

Question 2: Consider a Monte Carlo return $$G_t=R_{t+1}+f(R_{t+1}+f(R_{t+2}+f(...))).$$

Standard discounting can be seen as applying a linear transformation $$f(x) = \gamma x$$, by multiplying the remaining return after each step by a factor $$\gamma$$. Consider the following alternative where instead of multiplying with a factor $$\gamma$$, we raise the value to power: $$f(x)=x^{\gamma}$$. Does this mathematical model better explain the observed behaviour, in the sense that a $$γ$$ exists for which the optimal policy goes to $$+2$$? If so, give such a value for $$γ$$. If not, prove why not.

My answer to question 1: For the optimal policy to go to $$2$$, we need the return for going to $$+2$$ to be greater than both the return of going to $$+1$$ and $$+5$$, i.e., mathematically

\begin{align} 2\gamma > \gamma^25 \cap 2\gamma>1 \\ \frac{2}{5} > \gamma \cap \gamma > \frac{1}{2} \end{align}

Since $$(\frac{1}{2},\infty) \cap (-\infty, \frac{2}{5}) = \emptyset$$, this means that there is no such $$\gamma$$ for which the optimal policy is $$+2$$.

Again the return of $$+2$$ should be both greater than $$+1$$ and $$+5$$:

\begin{align} 2^{\gamma} > (5^{\gamma})^{\gamma} \cap 2^{\gamma}>1 \\ \gamma \ln 2 > \gamma^2 \ln 5 \cap \gamma > 0 \\ \frac{\ln 2}{\ln 5} > \gamma \cap \gamma > 0 \\ 0.43 > \gamma \cap \gamma > 0 \end{align}

Hence for $$0< \gamma < 0.43$$, the optimal policy will be to go to $$+2$$. Is this correct?

• Why are people voting to close this as opinion-based? So, now mathematics is an opinion?! What the...!
– nbro
Jul 7, 2022 at 16:12
• @nbro The 3 questions look very similar so I was confused thinking they are duplicates Jul 9, 2022 at 8:47
• @ArayKarjauv Ha, ok, don't worry then! ;)
– nbro
Jul 9, 2022 at 18:35
• Both look correct Aug 29, 2022 at 0:21