I'd like to ask you why, no matter my neural network function approximator for parametrized Q-learning implementation for a Contextual Bandits environment, I'm getting bad results. I don't know if it's a problem with my formulation of the problem and how I'm trying to solve it, or is it the neural architecture. I tried different fully-connected neural networks with different number of layers and different number of neurons (sticking to low numbers since my environment is not complex) but I always get bad results, and it seems the results are random.
if my implementation of the Q-learning algorithm for the Contextual Bandits problem is right. I made an environment that randomly generates three integers between 0 and 89 and given an action (integer between 0 and 4) it returns a reward following a certain logic (if all three integers are between 0 and 29 and the action is 0 then the reward is 0 otherwise it's -1).
My environment is:
class Environment():
def __init__(self):
self._observation = np.zeros((3,))
def interact(self, action):
self._observation = np.zeros((3,))
c1, c2, c3 = np.random.randint(0, 90, 3)
self._observation[0]=c1
self._observation[1]=c2
self._observation[2]=c3
reward = -1.0
condition = False
if (c1<30) and (c2<30) and (c3<30) and action==0:
condition = True
elif (30<=c1<60) and (30<=c2<60) and (30<=c3<60) and action==1:
condition = True
elif (60<=c1<90) and (60<=c2<90) and (60<=c3<90) and action==2:
condition = True
else:
if action==4:
condition = True
if condition:
reward = 0.0
return {"Observation": self._observation,
"Reward": reward}
The interaction method doesn't return state or time step, not like what TF-Agents environments' step method does. I just thought it's not necessary for the current problem; I don't rely on time steps since each state doesn't influence the next state. I thought that observation is what should be returned, the state being a more general data that could contain information the agent can't observe. I don't return the action too because we can get it outside the environment.
My function approximator of the Q-values are neural networks, always a fully connected architecture. For instance:
model = keras.models.Sequential([
keras.layers.Dense(16, activation="relu", input_shape=[n_inputs]),
keras.layers.Dense(16, activation="relu"),
keras.layers.Dense(n_outputs)])
I took the next blocks of code from Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow, 2nd Edition and adapted them to my situation:
env = Environment()
n_inputs = 3 #Observations are made of three integers
n_outputs = 4 #Four actions
def epsilon_greedy_policy(observation, epsilon=0):
if np.random.rand() < epsilon:
return np.random.randint(4)
else:
Q_values = model.predict(observation[np.newaxis])
return np.argmax(Q_values[0])
replay_buffer = deque(maxlen=2000)
def sample_experiences(batch_size):
indices = np.random.randint(len(replay_buffer), size=batch_size)
batch = [replay_buffer[index] for index in indices]
observations, rewards, actions = [np.array([experience[field_index] for experience in batch]) for field_index in range(3)]
return observations, rewards, actions
def play_one_step(env, observation, epsilon):
action = epsilon_greedy_policy(observation, epsilon)
observation, reward = env.interact(action).values()
replay_buffer.append((observation, reward, action))
return observation, reward
batch_size = 16
optimizer = keras.optimizers.Adam(learning_rate=1e-3)
loss_fn = keras.losses.mean_squared_error
def training_step(batch_size):
experiences = sample_experiences(batch_size)
observations, rewards, actions = experiences
target_Q_values = rewards
mask = tf.one_hot(actions, n_outputs)
with tf.GradientTape() as tape:
all_Q_values = model(observations)
Q_values = tf.reduce_sum(all_Q_values * mask, axis=1, keepdims=True)
loss = tf.reduce_mean(loss_fn(target_Q_values, Q_values))
grads = tape.gradient(loss, model.trainable_variables)
optimizer.apply_gradients(zip(grads, model.trainable_variables))
epsilon = 0.01
obs = np.random.randint(0,90,3)
for episode in tqdm(range(1000)):
if episode<250:
obs, reward = play_one_step(env, obs, epsilon)
else:
obs, reward = play_one_step(env, obs, epsilon)
training_step(batch_size)
I'm not sure at how to evaluate the performance of the agent, but I tried this as a first approach just to see if the predicted Q-values will enable a greedy-policy to choose the best action:
check0 = np.random.randint(0,30,3)
for i in range(30):
arr = np.random.randint(0,30,3)
check0 = np.vstack((check0, arr))
predictions = model.predict(check0)
c = 0
for i in range(predictions.shape[0]):
if np.argmax(predictions[i])==0:
c+=1
(c/predictions.shape[0])*100
Every time I ran the code above it gave me a totally different value. Sometimes it's 0%, sometimes it's 45%, sometimes it's 19%...
The issue is that no matter my model architecture, at the end, I get random results. I wonder if it's something wrong in the overall approach to solve the problem. I want to solve a Contextual Bandit where the agent observe a continuous context, take actions and try to link together the rewards obtained with the actions and the context in order to "understand" the logic behind it.
I hope you can help me figure out why do I get these random results.
Thank you.
