Reinforcement learning and Graph Neural Networks: Entropy drops to zero

I am currently working on an experiment to link reinforcement learning with graph neural networks. This is my architecture:

Feature Extraction with GCN:

• there is a fully meshed topology with 23 nodes. Therefore there are 23*22=506 edges.
• the original feature vector comprises 43 features that range from about -1 to 1.
• First, a neuronal network f takes calculates a vector per edge, given the source node and target node features.
• After we have calculated 506 edge vectors, function u aggregates the results from f per node (aggregation over 22 edges)
• A function g takes the original target feature vector and concatenates the aggregated results from u. Finally, the output dimension of g determines the new feature vector size for each node.
• At last, the function agg decides which information is returned from the feature extraction, e.g. just flatten the 23xg_output_dim feature vectors or building the average

After that:

• The output of the feature extractor is passed to the OpenAi Baseline PPO2 Implementation. The frameworks adds a flatten layer to the output and maps it to 19 action values and 1 value-function value.

I have made some observations in the experiments and do not manage to explain them. Hyperparameter are: an output dimension for f and g of 512. U=sum, aggr=flatten. A tanh activation is applied on the outputs of f and g. For PPO2: lr=0.000343, stepsize=512.

This gives me the following weight matrices:

<tf.Variable 'ppo2_model/pi/f_w_0:0' shape=(86, 512) dtype=float32_ref>
<tf.Variable 'ppo2_model/pi/g_w:0' shape=(555, 512) dtype=float32_ref>
<tf.Variable 'ppo2_model/pi/w:0' shape=(11776, 19) dtype=float32_ref>
<tf.Variable 'ppo2_model/vf/w:0' shape=(11776, 1) dtype=float32_ref>

The following problem occurs. Normally you wait for the entropy in the PPO2 to decrease during the training, because the algorithm learns which actions lead to more reward. With the described hyperparameters, the entropy drops abruptly to 0 within 100 update steps and stays zero even after >15.000 updates (=150M steps in the game). This means that the same action is always selected.

What I found out: the problem is that by making the sum over 22 edges, very large values are created (maximum 22*1 and 22*-1). The values are then given to the function g and thus ends up in the saturation region of the tanh. As a result, the new features of the 23 nodes contain many 1's and -1's. Because we flatten, the weighted sum of 11776 input neurons flows into each of the 19 action neurons, resulting in very large values in the policy. An action is then calculated from the policy with the following formula:

u = tf.random_uniform(tf.shape(logits), dtype=logits.dtype)
action = tf.argmax(logits - tf.log(-tf.log(u)), axis=-1),

Most of the time tf.log(-tf.log(u) gives sommething between 2 and -2 (in my opinion). This means that as soon as a very large value appears in the policy, the corresponding action is always selected and not the second or third most probable one, which might lead to more exploration.

What I don't understand 1): As soon as negative reward occurs, shouldn't the likelihood decrease again, so that in the end I choose other actions again?

I did some experiments with relu and elu activations: These are the value histogram of the output of g after the using relu, tanh and elu: These are the value histograms of the policy, when using relu, tanh and elu: Histogram over resulting actions:

What I don't understand 2:Using Relu you see that in the first steps in the policy were large values, but then the model learns to reduce the range, which is why in this example also the entropy does not drop. Why does this not work when using tanh or elu? We have found out 3 things with which the problem does not occur or is delayed. These are my assumptions. Are the correct in your opinion?:

• using smaller output dimension of f and g, like 6 or using aggr=mean -> For each of the 19 action neurons, less input neurons are averaged -> smaller values in the policy --> more exploration
• Using u=mean and not sum, averages the outputs of f, therefore the aggregated values are not only 1 and -1
• Smaller learning rate -> Making the weights too big, increases the chance of the 19 action values to be big. If there is no negativ reward, there is no need for the algorithm to make the weights smaller.

I know this is a lot of information, so I would be grateful for any small tip.!

• Hello. Could you please put your main specific question in the title? "... Issue with entropy" is not really a question but an announcement of some very general problem.
– nbro
Jan 25 '21 at 15:28
• Sorry for nitpicking a little, but that title can still be improved. It would be better if you asked the question in the form "Given situation X, why is the entropy dropping to zero"? You don't need to have "RL and GNNs" in the title, because they are already part of the tags.
– nbro
Jan 25 '21 at 16:43
• Please, share what is goal in your neural network and what is your input data. It will be helpful if you put structure of neural network as well. Mar 1 '21 at 22:53