0
$\begingroup$

So, I'm fairly new to reinforcement learning and I needed some help/explanations as to what the action_mask and avail_action fields alongside the action_embed_size actually mean in RLlib (the documentation for this library is not very beginner friendly/clear).

For an example, this is one of the resources (Action Masking With RLlib) I tried to use to help understand the above concepts. After reading the article, I completely understand what the action_mask does, but I'm still a bit confused as to what exactly the action_embed_size is and what the avail_actions fields actually are/represent (are the indices of avail_actions supposed to represent the action 0 if invalid, 1 if valid? Or are the elements supposed to represent the actions themselves - a value of 1, 4, 5, etc corresponding to the actual value of the action itself?).

Also when/how would there be a difference with the action_space and action_embed_size?

This is from the article that I used to sort of familiarize myself with the whole concept of Action Masking (this network is designed to solve the Knapsack Problem):

class KP0ActionMaskModel(TFModelV2):
    
    def __init__(self, obs_space, action_space, num_outputs,
        model_config, name, true_obs_shape=(11,),
        action_embed_size=5, *args, **kwargs):
        
        super(KP0ActionMaskModel, self).__init__(obs_space,
            action_space, num_outputs, model_config, name, 
            *args, **kwargs)
        
        self.action_embed_model = FullyConnectedNetwork(
            spaces.Box(0, 1, shape=true_obs_shape), 
                action_space, action_embed_size,
            model_config, name + "_action_embedding")
        self.register_variables(self.action_embed_model.variables())

    def forward(self, input_dict, state, seq_lens):
        avail_actions = input_dict["obs"]["avail_actions"]
        action_mask = input_dict["obs"]["action_mask"]
        action_embedding, _ = self.action_embed_model({
            "obs": input_dict["obs"]["state"]})
        intent_vector = tf.expand_dims(action_embedding, 1)
        action_logits = tf.reduce_sum(avail_actions * intent_vector,
            axis=1)
        inf_mask = tf.maximum(tf.log(action_mask), tf.float32.min)
        return action_logits + inf_mask, state

    def value_function(self):
        return self.action_embed_model.value_function()

From my understanding, the action_embedding is the output of the neural network and is then dotted with the action_mask to mask out illegal/invalid actions and finally passed to some kind of softmax function to get the final neural network output?

Please, correct me if I'm wrong.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.