I'm implementing a multilayer perceptron with Keras to predict the correct words order in a sentence.
I'm using train_on_batch()
because I convert each sentence in a tree and then order each local subtree: when each subtree is ordered, the entire tree is even ordered.
I notice a strange things during the training: since from the first epoch, the loss value is 0.0.
My dataset shape is (453732, 300)
(initially the number of features was 838, but I use PCA to reduce them) and this is the code:
mlp = keras.models.Sequential()
# add input layer
mlp.add(
keras.layers.Dense(
units=training_dataset.shape[1],
input_shape = (training_dataset.shape[1], ),
kernel_initializer='glorot_uniform',
bias_initializer='zeros',
activation='tanh')
)
# add hidden layer
mlp.add(
keras.layers.Dense(
units=training_dataset.shape[1] + 10,
input_shape = (training_dataset.shape[1] + 10,),
kernel_initializer='glorot_uniform',
bias_initializer='zeros',
activation='relu')
)
mlp.add(
keras.layers.Dropout(
0.2,
input_shape=(training_dataset.shape[1] + 10,))
)
# add output layer
mlp.add(
keras.layers.Dense(
units=1,
input_shape = (1, ),
kernel_initializer='glorot_uniform',
bias_initializer='zeros',
activation='softmax')
)
# define SGD optimizer
sgd_optimizer = keras.optimizers.SGD(
lr=0.01, decay=0.01, momentum=0.9, nesterov=True
)
# compile model
mlp.compile(
optimizer=sgd_optimizer,
loss=listnet_loss
)
mlp.summary() # print model settings
losses = np.array([])
# Training
with tf.device('/GPU:0'):
for epoch in range(0, 10):
print('Epoch {0} started!'.format(epoch))
start_range = 0
for group in groups_id_count:
end_range = (start_range + group[1]) # Batch is a group of words with same group id
batch_dataset = training_dataset[start_range:end_range, :]
batch_labels = training_dataset_labels[start_range:end_range]
batch_train_result = mlp.train_on_batch(batch_dataset, batch_labels)
losses = np.append(losses, batch_train_result)
start_range = end_range
print('Epoch {0} loss: {1}'.format(epoch, np.mean(losses)))
listnet_loss
is the following:
def get_top_one_probability(vector):
return (K.exp(vector) / K.sum(K.exp(vector)))
def listnet_loss(real_labels, predicted_labels):
return -K.sum(get_top_one_probability(real_labels)) * tf.math.log(get_top_one_probability(predicted_labels))
groups_id_count
is a list of tuple of the form (#subtree_number,#number_of_words_in_subtree)
, where #subtree_number
is an identifier for a subtree and #number_of_words_in_subtree
is the number of words in the subtree. Then, my batch are dynamic: a batch is composed by the number of words in a subtree. This is why I used train_on_batch()
to train my model.
Any suggestions? Thanks in advance.