2
$\begingroup$

I am trying to build an LSTM model to generate Shakspeare-like poems. I have training set $\{s_1,s_2, \dots,s_m\}$, which are sentences of Shakespeare poems, and each sentence contains words $\{w_1,w_2, \dots,w_n\}$.

To my understanding, each sentence $s_i$, for $i=1, \dots,m$ is a random sequence containing the words $w_j$, for $j=1, \dots,n$. The LSTM model is estimated by applying the maximum likelihood (MLE) method, which will use cross-entropy loss for optimization. The use of MLE requires that the samples in the random sequence be independent and identically distributed (i.i.d), however, the word sequence $w_j$ is not i.i.d (since it is non-Markov). Therefore, I am suspicious about using cross-entropy loss for training an LSTM for the NLP task (which seems to be the common practice).

Improve this question
$\endgroup$
3
  • $\begingroup$ The negative log-likelihood should indeed be equivalent to the cross-entropy, provided that the independence assumption is made. In fact, I have derived their equivalence in my notes by making this assumption. So, if you use the cross-entropy, you're implicitly making the independence assumption. $\endgroup$
    – nbro
    Commented Mar 8, 2020 at 2:03
  • $\begingroup$ Thanks for the edit and reply. I agree that negative log-likelihood is equivalent to cross-entropy when independence assumption is made. But for a NLP task, where the distribution for the next word is clearly not independent and identical to that of previous words, I am very suspicious on the adoption of cross-entropy loss. I wonder if there are any reasoning for using it or we can use any other loss for such task. $\endgroup$
    – Leey
    Commented Mar 9, 2020 at 3:18
  • $\begingroup$ Well, while training your model, you will maybe provide the sentences (or words) in random order, this should have a "decorrelation effect". However, I don't know if this is sufficient. I would need to review my notes about CE and NLL and where exactly do we assume independence. Also, in some cases (e.g. naive Bayes), the independence assumption is made (only for simplicity and because it sometimes works), while it often doesn't hold. $\endgroup$
    – nbro
    Commented Mar 9, 2020 at 3:26

0

Reset to default

You must log in to answer this question.