So, I'm fairly new to reinforcement learning and I needed some help/explanations as to what the action_mask and avail_action fields alongside the action_embed_size actually mean in RLlib (the documentation for this library is not very beginner friendly/clear).

For an example, this is one of the resources (Action Masking With RLlib) I tried to use to help understand the above concepts. After reading the article, I completely understand what the action_mask does, but I'm still a bit confused as to what exactly the action_embed_size is and what the avail_actions fields actually are/represent (are the indices of avail_actions supposed to represent the action 0 if invalid, 1 if valid? Or are the elements supposed to represent the actions themselves - a value of 1, 4, 5, etc corresponding to the actual value of the action itself?).

Also when/how would there be a difference with the action_space and action_embed_size?

This is from the article that I used to sort of familiarize myself with the whole concept of Action Masking (this network is designed to solve the Knapsack Problem):

class KP0ActionMaskModel(TFModelV2):
    def __init__(self, obs_space, action_space, num_outputs,
        model_config, name, true_obs_shape=(11,),
        action_embed_size=5, *args, **kwargs):
        super(KP0ActionMaskModel, self).__init__(obs_space,
            action_space, num_outputs, model_config, name, 
            *args, **kwargs)
        self.action_embed_model = FullyConnectedNetwork(
            spaces.Box(0, 1, shape=true_obs_shape), 
                action_space, action_embed_size,
            model_config, name + "_action_embedding")

    def forward(self, input_dict, state, seq_lens):
        avail_actions = input_dict["obs"]["avail_actions"]
        action_mask = input_dict["obs"]["action_mask"]
        action_embedding, _ = self.action_embed_model({
            "obs": input_dict["obs"]["state"]})
        intent_vector = tf.expand_dims(action_embedding, 1)
        action_logits = tf.reduce_sum(avail_actions * intent_vector,
        inf_mask = tf.maximum(tf.log(action_mask), tf.float32.min)
        return action_logits + inf_mask, state

    def value_function(self):
        return self.action_embed_model.value_function()

From my understanding, the action_embedding is the output of the neural network and is then dotted with the action_mask to mask out illegal/invalid actions and finally passed to some kind of softmax function to get the final neural network output?

Please, correct me if I'm wrong.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.