I am using the PPO algorithm implemented by tensorforce: https://github.com/reinforceio/tensorforce . It works great and I am very happy with the results.
However, I notice that there are many metaparameters available to give to the PPO algorithm:
# the tensorforce agent configuration ------------------------------------------
network_spec = [
dict(type='dense', size=256),
dict(type='dense', size=256),
]
agent = PPOAgent(
states=environment.states,
actions=environment.actions,
network=network_spec,
# Agent
states_preprocessing=None,
actions_exploration=None,
reward_preprocessing=None,
# MemoryModel
update_mode=dict(
unit='episodes',
# 10 episodes per update
batch_size=10,
# Every 10 episodes
frequency=10
),
memory=dict(
type='latest',
include_next_states=False,
capacity=200000
),
# DistributionModel
distributions=None,
entropy_regularization=0.01,
# PGModel
baseline_mode='states',
baseline=dict(
type='mlp',
sizes=[32, 32]
),
baseline_optimizer=dict(
type='multi_step',
optimizer=dict(
type='adam',
learning_rate=1e-3
),
num_steps=5
),
gae_lambda=0.97,
# PGLRModel
likelihood_ratio_clipping=0.2,
# PPOAgent
step_optimizer=dict(
type='adam',
learning_rate=1e-3
),
subsampling_fraction=0.2,
optimization_steps=25,
execution=dict(
type='single',
session_config=None,
distributed_spec=None
)
)
So my question is: is there a way to understand, intuitively, the meaning / effect of all these metaparameters and use this intuitive understanding to improve training performance?
So far I have reached - from a mix of reading the PPO paper and the literature around, and playing with the code - to the following conclusions. Can anybody complete / correct?
effect of network_spec: this is size of the 'main network'. Quite classical: need it big enough to get valuable predictions, not too big either otherwise it is hard to train.
effect of the parameters in update_mode: this is how often the network updates are performed.
batch_size is how many used for a batch update. Not sure of the effect neither what this exactly means in practice (are all samples taken from only 10 batches of the memory replay)?
frequency is how often the update is performed. I guess having frequency high would make the training slower but more stable (as sample from more different batches)?
unit: no idea what this does
memory: this is the replay memory buffer.
type: not sure what this does or how it works.
include_next_states: not sure what this does or how it works
capacity: I think this is how many tuples (state, action, reward) are stored. I think this is an important metaparameter. In my experience, if this is too low compared to the number of actions in one episode, the learning is very bad. I guess this is because it must be large enough to store MANY episodes, otherwise the network learns from correlated data - which is bad.
DistributionMode: guess this is the model for the distribution of the controls? No idea what the parameters there do.
PGModel: No idea what the paramaters there do. Would be interesting to know if some should be tweaked / which ones.
PGLRModel: idem, no idea what all these parameters do / if they should be tweaked.
PPOAgend: idem, no idea what all these parameters do / if they should be tweaked.
Summary
So in summary, would be great to get some help about:
- Which parameters should be tweaked
- How should these parameters be tweaked? Is there a 'high level intuition' about how they should be tweaked / in which circumstances?