In the paper, Contextual String Embeddings for Sequence Labeling, the authors state that

\begin{equation} P(x_{0:T}) = \prod_{t=0}^T P(x_t|x_{0:t-1}) \end{equation}

They also state that, in the LSTM architecture, the conditional probability $P(x_{t}|x_{0:t})$ is approximately a function of the network output $h_t$.

\begin{equation} P(x_{t}|x_{0:t}) \approx \prod_{t=0}^{T} P(x_t|h_t;\theta) \end{equation}

Why is this equation true?


In section 2.1 of the paper, the authors state that the goal of character-level language model is to estimate the following joint probability distribution

$$P(\boldsymbol{x}_{0:T}) = P(\boldsymbol{x}_{0}, \boldsymbol{x}_{1}, \dots, \boldsymbol{x}_{T}),$$

which is a joint probability distribution of all characters of a sequence of $T+1$ characters, where $\boldsymbol{x}_i$ is the $i$th character of the sequence. Ideally, the joint probability distribution, $P(\boldsymbol{x}_{0:T})$, should put more mass (or density) on the combination of $T-1$ characters that are more likely (in a given language). For example, in English, the combination of the characters "the" is more likely than the combination "bibliopole". Hence, $P(\text{t}, \text{h}, \text{e})$ should be higher than $P(\text{b}, \text{i}, \text{b}, \text{l}, \text{i}, \text{o}, \text{p}, \text{o}, \text{l}, \text{e})$, where, in this case, $T=2$. So, $P(\boldsymbol{x}_{0:T})$ is the actual character-level language model, which is represented by a joint probability distribution.

Similarly and intuitively, the conditional probability distribution $$P(\boldsymbol{x}_{t} \mid \boldsymbol{x}_{0:t-1})$$

tells us the probability of the next character of the sequence $\boldsymbol{x}_{t}$ given (that is, having observed) the previous characters of the sequence $\boldsymbol{x}_{0:t-1} = (\boldsymbol{x}_{0}, \dots, \boldsymbol{x}_{t-1})$. In general, the next character of a sequence $\boldsymbol{x}_{t}$ can depend on all previous characters of the same sequence, $\boldsymbol{x}_{0:t-1}$. If we are able to learn the conditional model $P(\boldsymbol{x}_{t} \mid \boldsymbol{x}_{0:t-1})$, given a sequence of $t$ characters, $\boldsymbol{x}_{0:t-1}$, we can sample the next character according to the same conditional probability distribution.

Recall that, given two events (or random variables) $A$ and $B$, the joint distribution of them is defined as $P(A, B) = P(A \mid B)P(B) = P(B \mid A)P(A)$, which gives rise to the Bayes' theorem. This can be easily generalised to multiple variables. More specifically, if you consider $B$ to a set of $N$ events (rather than just one), that is, $B=B_1 \cap B_2\cap \dots \cap B_N = B_1, B_2, \dots, B_N$, then $P(A, B) = P(A \mid B)P(B)$ still holds, but we can further decompose it

\begin{align} P(A, B) = P(A, B_1, B_2, \dots, B_N) &= P(A \mid B_1, B_2, \dots, B_N)P(B_1, B_2, \dots, B_N) \\ &= P(A \mid B_1, B_2, \dots, B_N)P(B_1 \mid B_2, \dots, B_N) P(B_2, \dots, B_N) \\ &= \cdots \end{align}

This is called the chain rule (or product rule) of probability. Essentially, we apply the rule $P(A, B) = P(A \mid B)P(B)$ recursively.

Analogously, in the paper, the authors apply this chain rule to express the joint distribution $P(\boldsymbol{x}_{0:T})$ as a product of conditional probability distributions, that is

$$ P(\boldsymbol{x}_{0:T}) = \prod_{t=0}^T P(\boldsymbol{x}_{t}\mid \boldsymbol{x}_{0:t-1}) $$

In the case of the LSTM, the vector $\boldsymbol{h}_t$ is supposed to keep track of the past. More specifically, in the case of the character-level language model, we train an LSTM-based RNN, so that $\boldsymbol{h}_t$ is approximately equal to $\boldsymbol{x}_{0:t-1}$, that is, $\boldsymbol{h}_t \approx \boldsymbol{x}_{0:t-1}$. Hence, the joint probability distribution of the characters above can be now be approximately defined as a function of the vector $\boldsymbol{h}_t$

$$ P(\boldsymbol{x}_{0:T}) \approx \prod_{t=0}^T P(\boldsymbol{x}_{t}\mid \boldsymbol{h}_t; \boldsymbol{\theta}) $$

where $\boldsymbol{\theta}$ are the parameters of the LSTM-based RNN. Note that this is not an equality but an approximation. Intuitively, we train the LSTM so that it learns the interactions of the past characters.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.