This question is similar to this question, however it has a different question.
I'm learning MDP's and I'd like to know if I'm doing these exercises correctly:
Consider the following MDP:
Suppose a behavioural scientist was doing an experiment where they gave rewards to an animal. Suppose $X=5$ (e.g., 5 food pellets at that location). It turns out that, after repeatedly exploring the grid, the animal seems to prefer going up to the rewards of $+2$.
Question 1: In this setting, prove that no scalar discount $\gamma\in[0,1]$ exists for which the optimal policy is to go to $+2$.
Question 2: Consider a Monte Carlo return $$G_t=R_{t+1}+f(R_{t+1}+f(R_{t+2}+f(...))).$$
Standard discounting can be seen as applying a linear transformation $f(x) = \gamma x$, by multiplying the remaining return after each step by a factor $\gamma$. Consider the following alternative where instead of multiplying with a factor $\gamma$, we raise the value to power: $f(x)=x^{\gamma}$. Does this mathematical model better explain the observed behaviour, in the sense that a $γ$ exists for which the optimal policy goes to $+2$? If so, give such a value for $γ$. If not, prove why not.
My answer to question 1: For the optimal policy to go to $2$, we need the return for going to $+2$ to be greater than both the return of going to $+1$ and $+5$, i.e., mathematically
\begin{align} 2\gamma > \gamma^25 \cap 2\gamma>1 \\ \frac{2}{5} > \gamma \cap \gamma > \frac{1}{2} \end{align}
Since $(\frac{1}{2},\infty) \cap (-\infty, \frac{2}{5}) = \emptyset$, this means that there is no such $\gamma$ for which the optimal policy is $+2$.
Answer to question 2:
Again the return of $+2$ should be both greater than $+1$ and $+5$:
\begin{align} 2^{\gamma} > (5^{\gamma})^{\gamma} \cap 2^{\gamma}>1 \\ \gamma \ln 2 > \gamma^2 \ln 5 \cap \gamma > 0 \\ \frac{\ln 2}{\ln 5} > \gamma \cap \gamma > 0 \\ 0.43 > \gamma \cap \gamma > 0 \end{align}
Hence for $0< \gamma < 0.43$, the optimal policy will be to go to $+2$. Is this correct?
