3
$\begingroup$

I implemented an LSTM with Keras to perform word ordering task (given a syntactically unordered sentence, the goal is to label each word of the sentence with the right position in this one.) So, my dataset is composed by numerical vectors and each numerical vector represents a word.

I train my model trying to learn the local order of a syntactic subtree composed by words that have syntactic relationships (for example, a subtree could be a set of three words in which the root is the verb and children are subject and object relationship).

I padded each subtree to a length of 20, which is the maximum subtree length that I found in my dataset. With padding introduction, I inserted a lot of vectors composed of only zeros.

My initial dataset shape is (700000, 837), but knowing that Keras wants a 3D dataset, I reshaped it to (35000, 20, 837) and the same for my labels (from 700000 to (35000, 20)).

As loss function, I'm using the ListNet algorithm loss function, which takes a list of words and for each computes the probability of the element to be ranked in the first position (then ranking these scores, I obtain the predicted labels of each word).

The current implementation is the following:

model = tf.keras.Sequential()
model.add(LSTM(units=100, activation='tanh', return_sequences=True, input_shape=(timesteps, features)))
model.add(Dense(1, activation='sigmoid'))

model.summary()

model.compile(loss=listnet_loss, optimizer=keras.optimizers.Adam(learning_rate=0.00005, beta_1=0.9, beta_2=0.999, amsgrad=True), metrics=["accuracy"])

model.fit(training_dataset, training_dataset_labels, batch_size=1, epochs=number_of_epochs, workers=10, verbose=1, callbacks=[SaveModelCallback()])

And SaveModelCallback simply saves each model during training.

At the moment I obtain, at each epoch, very very similar results:

Epoch 21/50
39200/39200 [==============================] - 363s 9ms/step - loss: 2.5483 - accuracy: 0.8246
Epoch 22/50
39200/39200 [==============================] - 359s 9ms/step - loss: 2.5480 - accuracy: 0.8245
Epoch 23/50
39200/39200 [==============================] - 360s 9ms/step - loss: 2.5478 - accuracy: 0.8246

I have to questions:

  1. Could zero-padding affect learning in a negative way? And if yes, how could we not consider this padding?

  2. Is it a good model for what I have to do?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.