1
$\begingroup$

I'd like to ask you why, no matter my neural network function approximator for parametrized Q-learning implementation for a Contextual Bandits environment, I'm getting bad results. I don't know if it's a problem with my formulation of the problem and how I'm trying to solve it, or is it the neural architecture. I tried different fully-connected neural networks with different number of layers and different number of neurons (sticking to low numbers since my environment is not complex) but I always get bad results, and it seems the results are random.

if my implementation of the Q-learning algorithm for the Contextual Bandits problem is right. I made an environment that randomly generates three integers between 0 and 89 and given an action (integer between 0 and 4) it returns a reward following a certain logic (if all three integers are between 0 and 29 and the action is 0 then the reward is 0 otherwise it's -1).

My environment is:

class Environment():

  def __init__(self):
      
      self._observation = np.zeros((3,))
  
  def interact(self, action):
      self._observation = np.zeros((3,))
      c1, c2, c3 = np.random.randint(0, 90, 3)
      self._observation[0]=c1
      self._observation[1]=c2
      self._observation[2]=c3
      reward = -1.0
      condition = False
      if (c1<30) and (c2<30) and (c3<30) and action==0:
          condition = True
      elif (30<=c1<60) and (30<=c2<60) and (30<=c3<60) and action==1:
          condition = True
      elif (60<=c1<90) and (60<=c2<90) and (60<=c3<90) and action==2:
          condition = True
      else:
          if action==4:
              condition = True
      if condition:
        reward = 0.0
            
      return {"Observation": self._observation,
                  "Reward": reward}

The interaction method doesn't return state or time step, not like what TF-Agents environments' step method does. I just thought it's not necessary for the current problem; I don't rely on time steps since each state doesn't influence the next state. I thought that observation is what should be returned, the state being a more general data that could contain information the agent can't observe. I don't return the action too because we can get it outside the environment.

My function approximator of the Q-values are neural networks, always a fully connected architecture. For instance:

model = keras.models.Sequential([
        keras.layers.Dense(16, activation="relu", input_shape=[n_inputs]),
        keras.layers.Dense(16, activation="relu"),
        keras.layers.Dense(n_outputs)])

I took the next blocks of code from Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow, 2nd Edition and adapted them to my situation:

env = Environment()

n_inputs = 3 #Observations are made of three integers
n_outputs = 4 #Four actions

def epsilon_greedy_policy(observation, epsilon=0):
  if np.random.rand() < epsilon:
    return np.random.randint(4)
  else:
    Q_values = model.predict(observation[np.newaxis])
    return np.argmax(Q_values[0])

replay_buffer = deque(maxlen=2000)

def sample_experiences(batch_size):
  indices = np.random.randint(len(replay_buffer), size=batch_size)
  batch = [replay_buffer[index] for index in indices]
  observations, rewards, actions = [np.array([experience[field_index] for experience in batch]) for field_index in range(3)]
  return observations, rewards, actions

def play_one_step(env, observation, epsilon):
  action = epsilon_greedy_policy(observation, epsilon)
  observation, reward = env.interact(action).values()
  replay_buffer.append((observation, reward, action))
  return observation, reward

batch_size = 16
optimizer = keras.optimizers.Adam(learning_rate=1e-3)
loss_fn = keras.losses.mean_squared_error

def training_step(batch_size):
  experiences = sample_experiences(batch_size)
  observations, rewards, actions = experiences
  target_Q_values = rewards
  mask = tf.one_hot(actions, n_outputs)
  with tf.GradientTape() as tape:
    all_Q_values = model(observations)
    Q_values = tf.reduce_sum(all_Q_values * mask, axis=1, keepdims=True)
    loss = tf.reduce_mean(loss_fn(target_Q_values, Q_values))
  grads = tape.gradient(loss, model.trainable_variables)
  optimizer.apply_gradients(zip(grads, model.trainable_variables))

epsilon = 0.01
obs = np.random.randint(0,90,3)
for episode in tqdm(range(1000)):
  if episode<250:
    obs, reward = play_one_step(env, obs, epsilon)
  else:
    obs, reward = play_one_step(env, obs, epsilon)
    training_step(batch_size)

I'm not sure at how to evaluate the performance of the agent, but I tried this as a first approach just to see if the predicted Q-values will enable a greedy-policy to choose the best action:

check0 = np.random.randint(0,30,3)

for i in range(30):
  arr = np.random.randint(0,30,3)
  check0 = np.vstack((check0, arr))

predictions = model.predict(check0)

c = 0
for i in range(predictions.shape[0]):
  if np.argmax(predictions[i])==0:
    c+=1

(c/predictions.shape[0])*100

Every time I ran the code above it gave me a totally different value. Sometimes it's 0%, sometimes it's 45%, sometimes it's 19%...

The issue is that no matter my model architecture, at the end, I get random results. I wonder if it's something wrong in the overall approach to solve the problem. I want to solve a Contextual Bandit where the agent observe a continuous context, take actions and try to link together the rewards obtained with the actions and the context in order to "understand" the logic behind it.

I hope you can help me figure out why do I get these random results.

Thank you.

$\endgroup$
3
$\begingroup$

Scale your neural network inputs.

The raw observations are in range $[0,89]$, and neural networks will cope badly with that used as inputs.

The ideal case for NN for each input feature is a gaussian distribution with mean 0, standard deviation 1. You don't need that to be perfect, though. A simple scale - divide each element by $30$ and subtract $1.5$ - will be fine here.

You can keep the environment as-is, and scale after observations are received. Up to you whether you put ready-scaled observations in the experience replay table or not. In your case it may be very slightly more efficient to do so in terms of CPU effort, but probably not something you would notice.

There are other ways you might deal with these kinds of numbers in a neural network's input, but pre-scaling to a standard range is usually the simplest and by far the most common solution.

$\endgroup$
5
  • 1
    $\begingroup$ Yeh the number 1 mistake a lot of beginners make (myself included!) was not normalizing their data. It makes an enormous difference $\endgroup$
    – Recessive
    Jul 9 at 4:03
  • $\begingroup$ Thank you for your answer. I applied the changes you suggested and while it improved the randomness of the predictions, it seems like the model now is only preferring one action over the others. So the maximum of the outputs will always be the fourth one, or always the first one... I tried to change the architecture again and that's how I got those different preferences but for one fixed architecture it's the same preference again and again. I wonder if I should make another post or just stick with the comment, but I'd like to ask you if you have any clue about what's going on. Thank you again $\endgroup$
    – Daviiid
    Jul 9 at 9:18
  • $\begingroup$ @Daviiid There is probably some other bug in your code. I just quickly scanned it and noticed the lack of scaling, which as Recessive points out is a common beginner's issue. Sorry I do not have time to look deeper at your code and find other implementation issues or bugs - it is very likely that you have something of that nature, where someone needs to work with you in more detail to fix the issue. If you are working alone, try to break the components apart (instead of one long script), and unit test them $\endgroup$ Jul 9 at 9:44
  • $\begingroup$ @NeilSlater Yes I fixed the scaling issue as you suggest. No need to look at the code in details thank you for guiding me, I'll look at the code in every detail. Should I remove the question since it's not really RL? $\endgroup$
    – Daviiid
    Jul 9 at 9:46
  • 1
    $\begingroup$ @Daviiid: IMO, your code contains one AI conceptual problem (not scaling) which is answered here, so meets the site goals. It gets tricky when there is more going on, but a lot of people do come here with first attempts at RL problems and scripts not quite working. There is some debate between AI SE admins about when a question like this crosses a line from being useful to being too noisy to help next person. But I think it is fine to keep this question. $\endgroup$ Jul 9 at 9:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.