I am studying for RL on my own and was trying to solve this question I came across.
Write an operator function $T(w, \pi, \mu, l, g)$ that takes weights $w$, a target policy $\pi$, a behaviour policy $\mu$, a trace parameter $l$, and a discount $g$, and outputs an off-policy-corrected lambda-return. For this question, implement the standard importance-weighted per-decision lambda-return. There will only be two actions, with the same policy in each state, so we can define $\pi$ to be a number which is the target probability of selecting action a in any state (s.t. $1 - \pi$ is the probability of selecting $b$), and similarly for the behaviour $\mu$.
Write an expected weight update, that uses the operator function $T$ and a value function $v$ to compute the expected weight update. The expectation should take into account the probabilities of actions in the future, as well as the steady-state (=long-term) probability of being in a state. The step size of the update should be $\alpha=0.1$.
Here is how my solution looks like (I am a total beginner in RL and in addition to studying Rich's book, I was trying to solve the basic intro course assignments as well to help understand the topic in detail.
x1 = np.array([1., 1.]) x2 = np.array([2., 1.]) def v(w, x): return x.T*w def T(w, pi, mu, l, g): states = [0, 1] n_states = len(states) #initial_dist = np.array([[1.0, 0.0]]) transition_matrix = np.array([[pi, 1-pi], [pi, 1-pi]]) if pi <= mu: # thresholding to select the state val = v(w, x1) else: val = v(w, x2) pi = 1 - pi l_power = np.power(l, n_states - 1) lambda_corrected = l_power * val lambda_corrected *= 1 - l return lambda_corrected - val def expected_update(w, pi, mu, l, g, lr): delta = T(w, pi, mu, l, g) w += lr * delta return w
The state diagram looks like this where there are two states $s_0$ and $s_1$. All rewards are $0$ and the state features $x_0 = x(s_0)$ and $x_1 = x(s_2)$ for two states are given as $x_1$ and $x_2$ in the code ([1., 1.], [2., 1.]) and also there are only two actions in each state $a$ and $b$. Action an always transitions to state $s_0$ (i.e. from s1 or from s0 itself) and action b always transitions to state $s_1$ (i.e. from $s_0$ or $s_1$ itself):
This is how the caller portion of the code looks like.
def caller(w, pi, mu, l, g): ws = [w] for _ in range(100): w = w + expected_update(w, pi, mu, l, g, lr=0.1) ws.append(w) return np.array(ws) mu = 0.2 # behaviour g = 0.99 # discount lambdas = np.array([0, 0.8, 0.9, 0.95, 1.]) pis = np.array([0., 0.1, 0.2, 0.5, 1.])
I would appreciate any help. Thanks.
Edit: I tried implementing the T() following the Bellman backup operator, but I am still not sure if I did this right or not.
return pi * g*v(w, x1) + (1-pi) * g*v(w, x2)