I am studying for RL on my own and was trying to solve this question I came across.

  1. Write an operator function $T(w, \pi, \mu, l, g)$ that takes weights $w$, a target policy $\pi$, a behaviour policy $\mu$, a trace parameter $l$, and a discount $g$, and outputs an off-policy-corrected lambda-return. For this question, implement the standard importance-weighted per-decision lambda-return. There will only be two actions, with the same policy in each state, so we can define $\pi$ to be a number which is the target probability of selecting action a in any state (s.t. $1 - \pi$ is the probability of selecting $b$), and similarly for the behaviour $\mu$.

  2. Write an expected weight update, that uses the operator function $T$ and a value function $v$ to compute the expected weight update. The expectation should take into account the probabilities of actions in the future, as well as the steady-state (=long-term) probability of being in a state. The step size of the update should be $\alpha=0.1$.

Here is how my solution looks like (I am a total beginner in RL and in addition to studying Rich's book, I was trying to solve the basic intro course assignments as well to help understand the topic in detail.

x1 = np.array([1., 1.])
x2 = np.array([2., 1.])

def v(w, x):
    return x.T*w

def T(w, pi, mu, l, g):
    states = [0, 1]
    n_states = len(states)
    #initial_dist = np.array([[1.0, 0.0]])
    transition_matrix = np.array([[pi, 1-pi],
                                  [pi, 1-pi]])
    if pi <= mu: # thresholding to select the state
        val = v(w, x1)
        val = v(w, x2)
        pi = 1 - pi

    l_power = np.power(l, n_states - 1)
    lambda_corrected = l_power * val
    lambda_corrected *= 1 - l

    return lambda_corrected - val

def expected_update(w, pi, mu, l, g, lr):
    delta = T(w, pi, mu, l, g)

    w += lr * delta
    return w

The state diagram looks like this where there are two states $s_0$ and $s_1$. All rewards are $0$ and the state features $x_0 = x(s_0)$ and $x_1 = x(s_2)$ for two states are given as $x_1$ and $x_2$ in the code ([1., 1.], [2., 1.]) and also there are only two actions in each state $a$ and $b$. Action an always transitions to state $s_0$ (i.e. from s1 or from s0 itself) and action b always transitions to state $s_1$ (i.e. from $s_0$ or $s_1$ itself): enter image description here

This is how the caller portion of the code looks like.

def caller(w, pi, mu, l, g):
  ws = [w]
  for _ in range(100):
    w = w + expected_update(w, pi, mu, l, g, lr=0.1)
  return np.array(ws)

mu = 0.2 # behaviour
g = 0.99  # discount

lambdas = np.array([0, 0.8, 0.9, 0.95, 1.])
pis = np.array([0., 0.1, 0.2, 0.5, 1.])

I would appreciate any help. Thanks.

Edit: I tried implementing the T() following the Bellman backup operator, but I am still not sure if I did this right or not.

return pi * g*v(w, x1) + (1-pi) * g*v(w, x2)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.