I am currently working on an experiment to link reinforcement learning with graph neural networks. This is my architecture:
Feature Extraction with GCN:
- there is a fully meshed topology with
23nodes. Therefore there are
- the original feature vector comprises
43features that range from about
- First, a neuronal network
ftakes calculates a vector per edge, given the source node and target node features.
- After we have calculated
506edge vectors, function
uaggregates the results from
fper node (aggregation over
- A function
gtakes the original target feature vector and concatenates the aggregated results from
u. Finally, the output dimension of
gdetermines the new feature vector size for each node.
- At last, the function
aggdecides which information is returned from the feature extraction, e.g. just flatten the
23xg_output_dimfeature vectors or building the average
- The output of the feature extractor is passed to the OpenAi Baseline PPO2 Implementation. The frameworks adds a
flattenlayer to the output and maps it to
19action values and
I have made some observations in the experiments and do not manage to explain them. Hyperparameter are: an output dimension for
tanh activation is applied on the outputs of
g. For PPO2:
This gives me the following weight matrices:
<tf.Variable 'ppo2_model/pi/f_w_0:0' shape=(86, 512) dtype=float32_ref> <tf.Variable 'ppo2_model/pi/g_w:0' shape=(555, 512) dtype=float32_ref> <tf.Variable 'ppo2_model/pi/w:0' shape=(11776, 19) dtype=float32_ref> <tf.Variable 'ppo2_model/vf/w:0' shape=(11776, 1) dtype=float32_ref>
The following problem occurs. Normally you wait for the entropy in the PPO2 to decrease during the training, because the algorithm learns which actions lead to more reward.
With the described hyperparameters, the entropy drops abruptly to
100 update steps and stays zero even after >
15.000 updates (=150M steps in the game). This means that the same action is always selected.
What I found out: the problem is that by making the sum over 22 edges, very large values are created (maximum
22*-1). The values are then given to the function
g and thus ends up in the saturation region of the tanh. As a result, the new features of the
23 nodes contain many
-1's. Because we flatten, the weighted sum of
11776 input neurons flows into each of the 19 action neurons, resulting in very large values in the policy. An action is then calculated from the policy with the following formula:
u = tf.random_uniform(tf.shape(logits), dtype=logits.dtype) action = tf.argmax(logits - tf.log(-tf.log(u)), axis=-1),
Most of the time
tf.log(-tf.log(u) gives sommething between 2 and -2 (in my opinion). This means that as soon as a very large value appears in the policy, the corresponding action is always selected and not the second or third most probable one, which might lead to more exploration.
What I don't understand 1): As soon as negative reward occurs, shouldn't the likelihood decrease again, so that in the end I choose other actions again?
I did some experiments with
These are the value histogram of the output of
g after the using
These are the value histograms of the policy, when using
Histogram over resulting actions:
What I don't understand 2:Using
Relu you see that in the first steps in the policy were large values, but then the model learns to reduce the range, which is why in this example also the entropy does not drop. Why does this not work when using
We have found out 3 things with which the problem does not occur or is delayed. These are my assumptions. Are the correct in your opinion?:
- using smaller output dimension of
aggr=mean-> For each of the 19 action neurons, less input neurons are averaged -> smaller values in the policy --> more exploration
u=meanand not sum, averages the outputs of
f, therefore the aggregated values are not only
- Smaller learning rate -> Making the weights too big, increases the chance of the 19 action values to be big. If there is no negativ reward, there is no need for the algorithm to make the weights smaller.
I know this is a lot of information, so I would be grateful for any small tip.!