So, I'm fairly new to reinforcement learning and I needed some help/explanations as to what the action_mask and avail_action fields alongside the action_embed_size actually mean in RLlib (the documentation for this library is not very beginner friendly/clear).
For an example, this is one of the resources (Action Masking With RLlib) I tried to use to help understand the above concepts. After reading the article, I completely understand what the action_mask does, but I'm still a bit confused as to what exactly the action_embed_size is and what the avail_actions fields actually are/represent (are the indices of avail_actions supposed to represent the action 0 if invalid, 1 if valid? Or are the elements supposed to represent the actions themselves - a value of 1, 4, 5, etc corresponding to the actual value of the action itself?).
Also when/how would there be a difference with the action_space and action_embed_size?
This is from the article that I used to sort of familiarize myself with the whole concept of Action Masking (this network is designed to solve the Knapsack Problem):
class KP0ActionMaskModel(TFModelV2):
def __init__(self, obs_space, action_space, num_outputs,
model_config, name, true_obs_shape=(11,),
action_embed_size=5, *args, **kwargs):
super(KP0ActionMaskModel, self).__init__(obs_space,
action_space, num_outputs, model_config, name,
*args, **kwargs)
self.action_embed_model = FullyConnectedNetwork(
spaces.Box(0, 1, shape=true_obs_shape),
action_space, action_embed_size,
model_config, name + "_action_embedding")
self.register_variables(self.action_embed_model.variables())
def forward(self, input_dict, state, seq_lens):
avail_actions = input_dict["obs"]["avail_actions"]
action_mask = input_dict["obs"]["action_mask"]
action_embedding, _ = self.action_embed_model({
"obs": input_dict["obs"]["state"]})
intent_vector = tf.expand_dims(action_embedding, 1)
action_logits = tf.reduce_sum(avail_actions * intent_vector,
axis=1)
inf_mask = tf.maximum(tf.log(action_mask), tf.float32.min)
return action_logits + inf_mask, state
def value_function(self):
return self.action_embed_model.value_function()
From my understanding, the action_embedding
is the output of the neural network and is then dotted with the action_mask
to mask out illegal/invalid actions and finally passed to some kind of softmax function to get the final neural network output?
Please, correct me if I'm wrong.