# Seq2Seq dialogs predicts only most common words like you after couple of epoches

I'm training Seq2Seq model on OpenSubtitles dialogs - Cornell-Movie-Dialogs-Corpus.

My work based on the following papers (but currently I'm not implemented Attention yet):

The loss I received is quite high and sucked in variation ~6.4 after 3 epoches. The model predicts the most common words with some times other not significant words (but 99.99% is just 'you'):

• I’ve experimented with 128 - 2048 hidden units and with 1 or 2 or 3 LSTM layers per encoder and decoder. The outcomes are more or less the same.

SEQ1: yeah man it means love respect community and the dollars too the package the unk end

SEQ2: but how did you get unk 82 end

PREDICTION: promoting 16th dashboard be of the the the you you you you you you you you you you you you you you you you you you you you you you you you

I'm using here greedy prediction, meaning - after I receive logit I do argmax(..) on all its value for first-3 mini-batch-elements (here I present only first element). For convenient - SEQ1 and SEQ2 are also printed - to know the actual dialog which was presented to the model.

The pseudo-code of my architecture looks like this (I'm using Tensorflow 1.5):

seq1 = tf.placeholder(...)
seq2 = tf.placeholder(...)

embeddings = tf.Variable(tf.random_uniform([vocab_size, 100],-1,1))

seq1_emb = tf.nn.embedding_lookup(embeddings, seq1)
seq2_emb = tf.nn.embedding_lookup(embeddings, seq1)

encoder_out, state1 = tf.nn.static_rnn(BasicLSTMCell(), seq1_emb)
decoder_out, state2 = tf.nn.static_rnn(BasicLSTMCell(), seq2_emb,
initial_state=state_1)
logit = Dense(decoder_out, use_bias=False)

crossent = tf.nn.saparse_softmax_cross_entropy_with_logits(logits=logit,
labels=target)
loss = tf.reduce_sum(crossent) / number_of_words_in_batch



I'm also wonder if I pass well state1 to decoder, which in general looks like this:

# reshape in pseudocode: state1 = state[1:]
new_state1 = []
for lstm in state1:
new_lstm = []
for gate in lstm:
new_lstm.append(gate[1:])
new_state1.append(tuple(new_lstm))
state1 = tuple(new_state1)

• Should I use some projection layer between states of encoder and decoder ?

So if seq1 has 32 words, seq2 has 31 (since we will not predict nothing after the last word, which is the tag <END>).