0
$\begingroup$

tldr; if I train the network on 1 training example, the outcome sometimes makes no sense at all, sometimes is as expected. If I train it on more examples and higher iterations, the network, which produces two outcomes (p and v) always predicts exactly 0 for v and I would like to change that.

In the following post I will provide all code necessary to reproduce the problem.
I am training a neural network on the same input. The wanted outcome for a value "v" is 1. If I create the network and train it, sometimes the predicted outcome will be 1, sometimes it will be -1.
Also, the loss seems to flip between 0 and 4 during training epochs.
Additionally, the loss blows up immensly, even though both losses for the outcome layers are close to zero.
I do not understand where this behaviour comes from. I used Leaky-ReLU to make sure it can handle negative input, I used a high learning rate to make sure the data in this example is sufficient on the training, and the input is the same all the time.

My Neural network looks like this:

input_layer = keras.Input(shape=(6,7),)    
formatted_input_layer = keras.layers.Reshape((6,7, 1))(input_layer)       
conv_layer1 = self.create_conv_layer(formatted_input_layer)
res_layer1 = self.create_res_layer(conv_layer1)
res_layer2 = self.create_res_layer(res_layer1)
res_layer3 = self.create_res_layer(res_layer2)
res_layer4 = self.create_res_layer(res_layer3)
policy_head = self.create_policy_head(res_layer4)
value_head = self.create_value_head(res_layer4)
model = keras.Model(inputs=input_layer, outputs=[policy_head, value_head])
optimizer = keras.optimizers.SGD(lr=args['lr'],momentum=args['momentum'])
model.compile(loss = {'policy_head' : 'categorical_crossentropy', 'value_head' : 'mean_squared_error'}, optimizer=optimizer, loss_weights={'policy_head':0.5, 'value_head':0.5})

Methods for the different layers:
conv_layer:

def create_conv_layer(self, input_layer):
    conv_layer = keras.layers.Conv2D(filters=256,
                                     kernel_size=3,
                                     strides=1,
                                     padding='same',
                                     use_bias=False,
                                     data_format="channels_last",
                                     activation = "linear",
                                     kernel_regularizer = keras.regularizers.l2(0.0001))(input_layer)
    conv_layer= keras.layers.BatchNormalization(axis=-1)(conv_layer)
    conv_layer = keras.layers.LeakyReLU()(conv_layer)
    return conv_layer

res_layer:

def create_res_layer(self, input_layer):
        conv_layer = self.create_conv_layer(input_layer)
        res_layer = keras.layers.Conv2D(filters=256,
                                         kernel_size=3,
                                         strides=1,
                                         padding='same',
                                         use_bias=False,
                                         data_format="channels_last",
                                         activation = "linear",
                                         kernel_regularizer = keras.regularizers.l2(0.0001))(conv_layer)
        res_layer= keras.layers.BatchNormalization(axis=-1)(res_layer)
        res_layer = keras.layers.add([input_layer, res_layer])
        res_layer = keras.layers.LeakyReLU()(res_layer)
        return res_layer

policy head:

def create_policy_head(self, input_layer):
        policy_head = keras.layers.Conv2D(filters=2,
                                          kernel_size=1,
                                          strides=1,
                                          padding='same',
                                          use_bias = False,
                                          data_format='channels_last',
                                          activation='linear',
                                          kernel_regularizer = keras.regularizers.l2(0.0001))(input_layer)
        policy_head = keras.layers.BatchNormalization(axis=-1)(policy_head)
        policy_head = keras.layers.LeakyReLU()(policy_head)
        policy_head = keras.layers.Flatten()(policy_head)
        policy_head = keras.layers.Dense(units = 7,
                                         use_bias = False,
                                         activation = 'softmax',
                                         kernel_regularizer = keras.regularizers.l2(0.0001),
                                         name = "policy_head"
                                         )(policy_head)
        return policy_head

value head:

def create_value_head(self, input_layer):
        value_head = keras.layers.Conv2D(filters=1,
                                          kernel_size=1,
                                          strides=1,
                                          padding='same',
                                          use_bias = False,
                                          data_format='channels_last',
                                          activation='linear',
                                          kernel_regularizer = keras.regularizers.l2(0.0001))(input_layer)
        value_head = keras.layers.BatchNormalization(axis=-1)(value_head)
        value_head = keras.layers.LeakyReLU()(value_head)  
        value_head = keras.layers.Flatten()(value_head)        
        value_head = keras.layers.Dense(units = 21,
                                         use_bias = False,
                                         activation = 'linear',
                                         kernel_regularizer = keras.regularizers.l2(0.0001)
                                         )(value_head)
        value_head = keras.layers.LeakyReLU()(value_head)      
        value_head = keras.layers.Dense(units = 1,
                                         use_bias = False,
                                         activation = 'tanh',
                                         kernel_regularizer = keras.regularizers.l2(0.0001),
                                         name = "value_head"                                     
                                         )(value_head)
        return value_head

                                )(value_head)

The way I am testing my NN:

    canonicalBoard = np.zeros(shape = (6,7), dtype=int) 


    Pi = [0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0] 

    trainExamples = [[canonicalBoard, Pi, 1]]*50        

    nnetwrapper.train(trainExamples)

    board = canonicalBoard[np.newaxis, :, :]

    p, v = nnetwrapper.nnet.model.predict(board)

which results in the training looking like this:

Epoch 1/10
50/50 [==============================] - 4s 71ms/step - loss: 1.6829 - policy_head_loss: 1.9459 - value_head_loss: 1.0000
Epoch 2/10
50/50 [==============================] - 1s 15ms/step - loss: 2.3218 - policy_head_loss: 3.8470 - value_head_loss: 0.3768
Epoch 3/10
50/50 [==============================] - 1s 13ms/step - loss: 4456112.5000 - policy_head_loss: 0.9027 - value_head_loss: 0.8510
Epoch 4/10
50/50 [==============================] - 1s 14ms/step - loss: 16085884.0000 - policy_head_loss: 0.0945 - value_head_loss: 3.9925
Epoch 5/10
50/50 [==============================] - 1s 14ms/step - loss: 32722448.0000 - policy_head_loss: 2.6572 - value_head_loss: 4.0000
Epoch 6/10
50/50 [==============================] - 1s 14ms/step - loss: 52690084.0000 - policy_head_loss: 9.6345 - value_head_loss: 3.1810e-12
Epoch 7/10
50/50 [==============================] - 1s 14ms/step - loss: 74703120.0000 - policy_head_loss: 1.1921e-07 - value_head_loss: 4.0000
Epoch 8/10
50/50 [==============================] - 1s 14ms/step - loss: 97784832.0000 - policy_head_loss: 1.1921e-07 - value_head_loss: 4.0000
Epoch 9/10
50/50 [==============================] - 1s 14ms/step - loss: 121202520.0000 - policy_head_loss: 2.0802e-05 - value_head_loss: 4.0000
Epoch 10/10
50/50 [==============================] - 1s 14ms/step - loss: 144415040.0000 - policy_head_loss: 1.1921e-07 - value_head_loss: 4.0000

and my prediction looking like this:

p: [[0. 0. 0. 0. 0. 1. 0.]]  v: [[-1.]]

another outcome could be:

Epoch 1/10
50/50 [==============================] - 4s 82ms/step - loss: 1.6829 - policy_head_loss: 1.9459 - value_head_loss: 1.0000
Epoch 2/10
50/50 [==============================] - 1s 17ms/step - loss: 2.2826 - policy_head_loss: 2.0001 - value_head_loss: 2.1454
Epoch 3/10
50/50 [==============================] - 1s 16ms/step - loss: 1718694.1250 - policy_head_loss: 0.5434 - value_head_loss: 3.9772
Epoch 4/10
50/50 [==============================] - 1s 14ms/step - loss: 6204218.0000 - policy_head_loss: 9.4180e-05 - value_head_loss: 0.0000e+00
Epoch 5/10
50/50 [==============================] - 1s 14ms/step - loss: 12620835.0000 - policy_head_loss: 1.1921e-07 - value_head_loss: 0.0000e+00
Epoch 6/10
50/50 [==============================] - 1s 14ms/step - loss: 20322232.0000 - policy_head_loss: 7.7489 - value_head_loss: 0.0000e+00
Epoch 7/10
50/50 [==============================] - 1s 14ms/step - loss: 28812526.0000 - policy_head_loss: 1.1921e-07 - value_head_loss: 1.5966
Epoch 8/10
50/50 [==============================] - 1s 14ms/step - loss: 37715012.0000 - policy_head_loss: 1.1921e-07 - value_head_loss: 0.0000e+00
Epoch 9/10
50/50 [==============================] - 1s 15ms/step - loss: 46747064.0000 - policy_head_loss: 1.1921e-07 - value_head_loss: 0.0000e+00
Epoch 10/10
50/50 [==============================] - 1s 14ms/step - loss: 55699992.0000 - policy_head_loss: 1.1921e-07 - value_head_loss: 0.0000e+00

with the predictions:

p: [[0. 0. 0. 0. 0. 1. 0.]]  v: [[1.]]

which are the correct ones as I would have expected.

How come on some trainings, my NN doesnt fit the data at all? I wanted to start the training process for a whole week now, but before I do so I want to make sure there are no errors in the way I layed out my NN. And this looks like I am missing something here.

And here at the predict / train methods of my neuralnet, (the NN is part of an alpha-zero replica and the involved game is connect4, I omitted these in the example to make it easier to actually replicate the problem. This is why you see some transform operations in predict and train methods)

def train(self, examples):
        input_boards, target_pis, target_vs = list(zip(*examples))       
        input_boards = np.asarray(input_boards)
        target_pis = np.asarray(target_pis)
        target_vs = np.asarray(target_vs)        
        logger.debug("Passing to nn: x: {}, y: {}, batch_size: {}, epochs: {}".format(input_boards, [target_pis, target_vs], self.args["batch_size"], self.args["epochs"]))
        self.nnet.model.fit(x = input_boards, y = [target_pis, target_vs], batch_size = self.args["batch_size"], epochs = self.args["epochs"])


def predict(self, board):
        board = board.nn_board_2d
        # preparing input
        board = board[np.newaxis, :, :] # this has to be done for the conv2d to work

        # run
        pi, v = self.nnet.model.predict(board)
        return pi[0], v[0]

The parameters I used for this example:

'lr': 0.2
'dropout': 0.1
'epochs': 10
'num_channels': 512,
'filters': 256
'momentum':0.9

EDIT: As soon as I use a lower learning rate and more iterations, my p changes, but my v stays exactly at 0. This is what was bothering me in the first place:

Epoch 1/10
118/118 [==============================] - 5s 44ms/step - loss: 1.8037 - policy_head_loss: 2.4305 - value_head_loss: 0.7578
Epoch 2/10
118/118 [==============================] - 2s 14ms/step - loss: 1.1666 - policy_head_loss: 1.7723 - value_head_loss: 0.1416
Epoch 3/10
118/118 [==============================] - 2s 13ms/step - loss: 1.0987 - policy_head_loss: 1.6832 - value_head_loss: 0.0948
Epoch 4/10
118/118 [==============================] - 2s 13ms/step - loss: 1.0430 - policy_head_loss: 1.5787 - value_head_loss: 0.0876
Epoch 5/10
118/118 [==============================] - 2s 13ms/step - loss: 0.9943 - policy_head_loss: 1.4859 - value_head_loss: 0.0826
Epoch 6/10
118/118 [==============================] - 2s 13ms/step - loss: 0.9469 - policy_head_loss: 1.3959 - value_head_loss: 0.0777
Epoch 7/10
118/118 [==============================] - 2s 13ms/step - loss: 0.9046 - policy_head_loss: 1.3180 - value_head_loss: 0.0707
Epoch 8/10
118/118 [==============================] - 2s 13ms/step - loss: 0.8629 - policy_head_loss: 1.2403 - value_head_loss: 0.0647
Epoch 9/10
118/118 [==============================] - 2s 14ms/step - loss: 0.8068 - policy_head_loss: 1.1344 - value_head_loss: 0.0583
Epoch 10/10
118/118 [==============================] - 2s 13ms/step - loss: 0.7335 - policy_head_loss: 0.9922 - value_head_loss: 0.0538
I0916 14:24:38.595780  7116 trainingonly1NN.py:232] Values for empty board: new network: P : [0.20057818 0.11068489 0.15129891 0.20042823 0.13117987 0.04705378
 0.15877616]  v: 0
$\endgroup$
  • 1
    $\begingroup$ You misunderstood about the learning rate. Your network is diverging. Use a low learning rate (and lots of iterations) instead. $\endgroup$ – Neil Slater Sep 16 at 12:04
  • $\begingroup$ But both losses I am tracking (policy and value, these are output units) reach 0, how can the overall loss go up? I wonder if there is a weight initiliazation issue or something $\endgroup$ – Leviathan Sep 16 at 12:08
  • $\begingroup$ Sorry, I missed the combined nature of the losses. I'm not sure what is going on. However your premise "I used a high learning rate to make sure the data in this example is sufficient on the training" is wrong in general. High learning rates will often lead to diverging loss values. What happens when you try a lower learning rate and more iterations? $\endgroup$ – Neil Slater Sep 16 at 14:55
  • $\begingroup$ Then P does get updated correctly, but v stays at 0. This is what is so odd to me. I mean it stays exactly 0, not 0.0001 or -0.0001 or something, it always stays at 0. and that is pretty bad $\endgroup$ – Leviathan Sep 16 at 15:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.