0
$\begingroup$

My framework is an encoder-decoder (LSTM-to-LSTM) model, similar to this post. The model basically reads a sentence and generate another sentence. But, the thing is, after a few epochs training, the model cannot produce any meaningful output, instead it keeps generating repeating words.

The framework can translate Franch-Enlgish very effectively, but for my problem it generates the result like this.

Can you explain why it produces such result, Thank you

  • Here is my printed output:

The screenshot from the output

$\endgroup$
2
  • $\begingroup$ Hello. Although not strictly necessary in this case, rather than providing a screenshot of the output, it would be better if you copy and paste the output. You should also provide more details about your model, how you trained it, etc., because saying that your model is similar to another may not be sufficient to fully answer your question. $\endgroup$ – nbro Apr 26 at 0:59
  • $\begingroup$ Yes your are right, I would update it after I finish the experiment. $\endgroup$ – Cheleeger Ken Apr 26 at 1:02
1
$\begingroup$

The trained model predicts the probability of a given sequence of tokens. Whatever NLP task you are doing, you usually want to get a high-probability sample from that probability distribution. This sampling task could be quite non-trivial.

What you are seeing is most likely the result of a greedy sampling - the most probable next word is chosen from the probability distribution. This quite often leads to infinite repletion loops as you experience.

Simplest way to solve the repetition problem is to actually randomly sample from the probability distribution of the next word - the distribution is usually augmented with so-called temperature parameter.

Finally, there's beam search - you just perform a tree search for a best N-gram sample in the future. That's the most sophisticated, and the most computationally hard one.

$\endgroup$
1
  • $\begingroup$ Thanks, I will try your recommendation. $\endgroup$ – Cheleeger Ken Apr 26 at 1:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.