I implemented an
Keras to perform word ordering task (given a syntactically unordered sentence, the goal is to label each word of the sentence with the right position in this one.)
So, my dataset is composed by numerical vectors and each numerical vector represents a word.
I train my model trying to learn the local order of a syntactic subtree composed by words that have syntactic relationships (for example, a subtree could be a set of three words in which the root is the verb and children are subject and object relationship).
I padded each subtree to a length of 20, which is the maximum subtree length that I found in my dataset. With padding introduction, I inserted a lot of vectors composed of only zeros.
My initial dataset shape is
(700000, 837), but knowing that
Keras wants a 3D dataset, I reshaped it to
(35000, 20, 837) and the same for my labels (from 700000 to
As loss function, I'm using the
ListNet algorithm loss function, which takes a list of words and for each computes the probability of the element to be ranked in the first position (then ranking these scores, I obtain the predicted labels of each word).
The current implementation is the following:
model = tf.keras.Sequential() model.add(LSTM(units=100, activation='tanh', return_sequences=True, input_shape=(timesteps, features))) model.add(Dense(1, activation='sigmoid')) model.summary() model.compile(loss=listnet_loss, optimizer=keras.optimizers.Adam(learning_rate=0.00005, beta_1=0.9, beta_2=0.999, amsgrad=True), metrics=["accuracy"]) model.fit(training_dataset, training_dataset_labels, batch_size=1, epochs=number_of_epochs, workers=10, verbose=1, callbacks=[SaveModelCallback()])
SaveModelCallback simply saves each model during training.
At the moment I obtain, at each epoch, very very similar results:
Epoch 21/50 39200/39200 [==============================] - 363s 9ms/step - loss: 2.5483 - accuracy: 0.8246 Epoch 22/50 39200/39200 [==============================] - 359s 9ms/step - loss: 2.5480 - accuracy: 0.8245 Epoch 23/50 39200/39200 [==============================] - 360s 9ms/step - loss: 2.5478 - accuracy: 0.8246
I have to questions:
Could zero-padding affect learning in a negative way? And if yes, how could we not consider this padding?
Is it a good model for what I have to do?