Tuning of PPO metaparameters: a high level overview of what each parameter does

Question

I am using the PPO algorithm implemented by tensorforce: https://github.com/reinforceio/tensorforce . It works great and I am very happy with the results.

However, I notice that there are many metaparameters available to give to the PPO algorithm:

 # the tensorforce agent configuration ------------------------------------------
    network_spec = [
        dict(type='dense', size=256),
        dict(type='dense', size=256),
    ]

    agent = PPOAgent(
        states=environment.states,
        actions=environment.actions,
        network=network_spec,
        # Agent
        states_preprocessing=None,
        actions_exploration=None,
        reward_preprocessing=None,
        # MemoryModel
        update_mode=dict(
            unit='episodes',
            # 10 episodes per update
            batch_size=10,
            # Every 10 episodes
            frequency=10
        ),
        memory=dict(
            type='latest',
            include_next_states=False,
            capacity=200000
        ),
        # DistributionModel
        distributions=None,
        entropy_regularization=0.01,
        # PGModel
        baseline_mode='states',
        baseline=dict(
            type='mlp',
            sizes=[32, 32]
        ),
        baseline_optimizer=dict(
            type='multi_step',
            optimizer=dict(
                type='adam',
                learning_rate=1e-3
            ),
            num_steps=5
        ),
        gae_lambda=0.97,
        # PGLRModel
        likelihood_ratio_clipping=0.2,
        # PPOAgent
        step_optimizer=dict(
            type='adam',
            learning_rate=1e-3
        ),
        subsampling_fraction=0.2,
        optimization_steps=25,
        execution=dict(
            type='single',
            session_config=None,
            distributed_spec=None
        )
    )

So my question is: is there a way to understand, intuitively, the meaning / effect of all these metaparameters and use this intuitive understanding to improve training performance?

So far I have reached - from a mix of reading the PPO paper and the literature around, and playing with the code - to the following conclusions. Can anybody complete / correct?

• effect of network_spec: this is size of the 'main network'. Quite classical: need it big enough to get valuable predictions, not too big either otherwise it is hard to train.

• effect of the parameters in update_mode: this is how often the network updates are performed.

  • batch_size is how many used for a batch update. Not sure of the effect neither what this exactly means in practice (are all samples taken from only 10 batches of the memory replay)?

  • frequency is how often the update is performed. I guess having frequency high would make the training slower but more stable (as sample from more different batches)?

  • unit: no idea what this does

• memory: this is the replay memory buffer.

  • type: not sure what this does or how it works.

  • include_next_states: not sure what this does or how it works

  • capacity: I think this is how many tuples (state, action, reward) are stored. I think this is an important metaparameter. In my experience, if this is too low compared to the number of actions in one episode, the learning is very bad. I guess this is because it must be large enough to store MANY episodes, otherwise the network learns from correlated data - which is bad.

• DistributionMode: guess this is the model for the distribution of the controls? No idea what the parameters there do.

• PGModel: No idea what the paramaters there do. Would be interesting to know if some should be tweaked / which ones.

• PGLRModel: idem, no idea what all these parameters do / if they should be tweaked.

• PPOAgend: idem, no idea what all these parameters do / if they should be tweaked.

Summary

So in summary, would be great to get some help about:

• Which parameters should be tweaked
• How should these parameters be tweaked? Is there a 'high level intuition' about how they should be tweaked / in which circumstances?

Answer 1

Some investigation about the memory dict: The current type is latest, which means you're not using a memory replay, but a latest memory. Switching to replay may help. Also, include_next_state means that you store tuples (state, action, reward, next state). It's not a real parameter though, because in PPO it must be set to False, otherwise an error is raised. Your interpretation of capacity looks OK.

About the update mode spec dict, your current settings mean that : every 10 (frequency) episodes (unit), you pull a batch of 10 (batch_size) episodes (unit) from the memory (the pulling method is defined through the memory dict), and you perform an optimization step over this batch. Be aware that the unit defines both the unit of the optimization frequency and the type of the object fetched from the memory.