# Off-policy Bellman Operators: Writing Operator and Weight Update Function for a 2-State System

I am studying for RL on my own and was trying to solve this question I came across.

1. Write an operator function $$T(w, \pi, \mu, l, g)$$ that takes weights $$w$$, a target policy $$\pi$$, a behaviour policy $$\mu$$, a trace parameter $$l$$, and a discount $$g$$, and outputs an off-policy-corrected lambda-return. For this question, implement the standard importance-weighted per-decision lambda-return. There will only be two actions, with the same policy in each state, so we can define $$\pi$$ to be a number which is the target probability of selecting action a in any state (s.t. $$1 - \pi$$ is the probability of selecting $$b$$), and similarly for the behaviour $$\mu$$.

2. Write an expected weight update, that uses the operator function $$T$$ and a value function $$v$$ to compute the expected weight update. The expectation should take into account the probabilities of actions in the future, as well as the steady-state (=long-term) probability of being in a state. The step size of the update should be $$\alpha=0.1$$.

Here is how my solution looks like (I am a total beginner in RL and in addition to studying Rich's book, I was trying to solve the basic intro course assignments as well to help understand the topic in detail.

x1 = np.array([1., 1.])
x2 = np.array([2., 1.])

def v(w, x):
return x.T*w

def T(w, pi, mu, l, g):
states = [0, 1]
n_states = len(states)
#initial_dist = np.array([[1.0, 0.0]])
transition_matrix = np.array([[pi, 1-pi],
[pi, 1-pi]])

if pi <= mu: # thresholding to select the state
val = v(w, x1)
else:
val = v(w, x2)
pi = 1 - pi

l_power = np.power(l, n_states - 1)
lambda_corrected = l_power * val
lambda_corrected *= 1 - l

return lambda_corrected - val

def expected_update(w, pi, mu, l, g, lr):
delta = T(w, pi, mu, l, g)

w += lr * delta
return w


The state diagram looks like this where there are two states $$s_0$$ and $$s_1$$. All rewards are $$0$$ and the state features $$x_0 = x(s_0)$$ and $$x_1 = x(s_2)$$ for two states are given as $$x_1$$ and $$x_2$$ in the code ([1., 1.], [2., 1.]) and also there are only two actions in each state $$a$$ and $$b$$. Action an always transitions to state $$s_0$$ (i.e. from s1 or from s0 itself) and action b always transitions to state $$s_1$$ (i.e. from $$s_0$$ or $$s_1$$ itself):

This is how the caller portion of the code looks like.

def caller(w, pi, mu, l, g):
ws = [w]
for _ in range(100):
w = w + expected_update(w, pi, mu, l, g, lr=0.1)
ws.append(w)
return np.array(ws)

mu = 0.2 # behaviour
g = 0.99  # discount

lambdas = np.array([0, 0.8, 0.9, 0.95, 1.])
pis = np.array([0., 0.1, 0.2, 0.5, 1.])


I would appreciate any help. Thanks.

Edit: I tried implementing the T() following the Bellman backup operator, but I am still not sure if I did this right or not.

return pi * g*v(w, x1) + (1-pi) * g*v(w, x2)