I'm training Seq2Seq model on OpenSubtitles dialogs - Cornell-Movie-Dialogs-Corpus.
My work based on the following papers (but currently I'm not implemented Attention yet):
- Sequence to Sequence Learning with Neural Networks, Sutskever et al. 2014
- A Neural Conversational Model, Vinyals, Le, 2015
loss I received is quite high and sucked in variation
~6.4 after 3 epoches. The model predicts the most common words with some times other not significant words (but 99.99% is just 'you'):
- I’ve experimented with 128 - 2048 hidden units and with 1 or 2 or 3 LSTM layers per
decoder. The outcomes are more or less the same.
SEQ1: yeah man it means love respect community and the dollars too the package the unk end
SEQ2: but how did you get unk 82 end
PREDICTION: promoting 16th dashboard be of the the the you you you you you you you you you you you you you you you you you you you you you you you you
I'm using here
greedy prediction, meaning - after I receive
logit I do
argmax(..) on all its value for first-3 mini-batch-elements (here I present only first element). For convenient -
SEQ2 are also printed - to know the actual dialog which was presented to the model.
The pseudo-code of my architecture looks like this (I'm using Tensorflow 1.5):
seq1 = tf.placeholder(...) seq2 = tf.placeholder(...) embeddings = tf.Variable(tf.random_uniform([vocab_size, 100],-1,1)) seq1_emb = tf.nn.embedding_lookup(embeddings, seq1) seq2_emb = tf.nn.embedding_lookup(embeddings, seq1) encoder_out, state1 = tf.nn.static_rnn(BasicLSTMCell(), seq1_emb) decoder_out, state2 = tf.nn.static_rnn(BasicLSTMCell(), seq2_emb, initial_state=state_1) logit = Dense(decoder_out, use_bias=False) crossent = tf.nn.saparse_softmax_cross_entropy_with_logits(logits=logit, labels=target) crossent = mask_padded_zeros(crossent) loss = tf.reduce_sum(crossent) / number_of_words_in_batch train = tf.train.AdamOptimizer(learning_rate=0.00002).minimize(loss)
I'm also wonder if I pass well
decoder, which in general looks like this:
# reshape in pseudocode: state1 = state[1:] new_state1 =  for lstm in state1: new_lstm =  for gate in lstm: new_lstm.append(gate[1:]) new_state1.append(tuple(new_lstm)) state1 = tuple(new_state1)
- Should I use some projection layer between states of
seq1 has 32 words,
seq2 has 31 (since we will not predict nothing after the last word, which is the tag