I am writing a AlphaGo Zero clone, and sometimes in the training the policy head loss and value head loss would both be decreasing, but the total loss is increasing?
How is this possible? I am using Adam, and here's the code:
input_tensor = Input((self.input_board_size, self.input_board_size, 2))
x = input_tensor
x = self.convolution_block(x)
for _ in range(self.number_of_residual_block):
x = self.residual_block(x)
self.model = Model(inputs=input_tensor, outputs=[self.policy_head(x), self.value_head(x)])['categorical_crossentropy', 'mean_squared_error'])
self.model.compile(Adam(lr=1e-4), ['categorical_crossentropy', 'mean_squared_error'])