I'm training Seq2Seq model on OpenSubtitles dialogs - Cornell-Movie-Dialogs-Corpus.

My work based on the following papers (but currently I'm not implemented Attention yet):

The loss I received is quite high and sucked in variation ~6.4 after 3 epoches. The model predicts the most common words with some times other not significant words (but 99.99% is just 'you'):

  • I’ve experimented with 128 - 2048 hidden units and with 1 or 2 or 3 LSTM layers per encoder and decoder. The outcomes are more or less the same.

SEQ1: yeah man it means love respect community and the dollars too the package the unk end

SEQ2: but how did you get unk 82 end

PREDICTION: promoting 16th dashboard be of the the the you you you you you you you you you you you you you you you you you you you you you you you you

I'm using here greedy prediction, meaning - after I receive logit I do argmax(..) on all its value for first-3 mini-batch-elements (here I present only first element). For convenient - SEQ1 and SEQ2 are also printed - to know the actual dialog which was presented to the model.

The pseudo-code of my architecture looks like this (I'm using Tensorflow 1.5):

seq1 = tf.placeholder(...)
seq2 = tf.placeholder(...)

embeddings = tf.Variable(tf.random_uniform([vocab_size, 100],-1,1))

seq1_emb = tf.nn.embedding_lookup(embeddings, seq1)
seq2_emb = tf.nn.embedding_lookup(embeddings, seq1)

encoder_out, state1 = tf.nn.static_rnn(BasicLSTMCell(), seq1_emb)
decoder_out, state2 = tf.nn.static_rnn(BasicLSTMCell(), seq2_emb,
logit = Dense(decoder_out, use_bias=False)

crossent = tf.nn.saparse_softmax_cross_entropy_with_logits(logits=logit, 
crossent = mask_padded_zeros(crossent)
loss = tf.reduce_sum(crossent) / number_of_words_in_batch

train = tf.train.AdamOptimizer(learning_rate=0.00002).minimize(loss) 

I'm also wonder if I pass well state1 to decoder, which in general looks like this:

# reshape in pseudocode: state1 = state[1:]
new_state1 = []
for lstm in state1:
    new_lstm = []
    for gate in lstm:
state1 = tuple(new_state1)
  • Should I use some projection layer between states of encoder and decoder ?

So if seq1 has 32 words, seq2 has 31 (since we will not predict nothing after the last word, which is the tag <END>).


1 Answer 1


To answer my own question - it was because of 2 things:

  • Too small number of batches - Model just started to gain statistical knowledge about language dialogs. I needed to train it longer.

  • I masked sequence too early (wrongly removed the < END > tag) - Because on each sentence the last world is just < END > tag - I removed it on all training examples, which prevented Model from learn what does it mean "the end of the sentence".

The last condition probably caused that strange pattern even further, because if model doesn't know what word to put in (and because of the lack of the < END > tag) it must fill each sentence till the end of max_sequence_len.

So the Model inputed in loop-like-manner - one of the most common word (where there was no signal from target sentence, because it simply ended).


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .