# Why can we approximate the joint probability distribution using the output vector of an LSTM?

In the paper, Contextual String Embeddings for Sequence Labeling, the authors state that

$$$$P(x_{0:T}) = \prod_{t=0}^T P(x_t|x_{0:t-1})$$$$

They also state that, in the LSTM architecture, the conditional probability $$P(x_{t}|x_{0:t})$$ is approximately a function of the network output $$h_t$$.

$$$$P(x_{t}|x_{0:t}) \approx \prod_{t=0}^{T} P(x_t|h_t;\theta)$$$$

Why is this equation true?

In section 2.1 of the paper, the authors state that the goal of character-level language model is to estimate the following joint probability distribution

$$P(\boldsymbol{x}_{0:T}) = P(\boldsymbol{x}_{0}, \boldsymbol{x}_{1}, \dots, \boldsymbol{x}_{T}),$$

which is a joint probability distribution of all characters of a sequence of $$T+1$$ characters, where $$\boldsymbol{x}_i$$ is the $$i$$th character of the sequence. Ideally, the joint probability distribution, $$P(\boldsymbol{x}_{0:T})$$, should put more mass (or density) on the combination of $$T-1$$ characters that are more likely (in a given language). For example, in English, the combination of the characters "the" is more likely than the combination "bibliopole". Hence, $$P(\text{t}, \text{h}, \text{e})$$ should be higher than $$P(\text{b}, \text{i}, \text{b}, \text{l}, \text{i}, \text{o}, \text{p}, \text{o}, \text{l}, \text{e})$$, where, in this case, $$T=2$$. So, $$P(\boldsymbol{x}_{0:T})$$ is the actual character-level language model, which is represented by a joint probability distribution.

Similarly and intuitively, the conditional probability distribution $$P(\boldsymbol{x}_{t} \mid \boldsymbol{x}_{0:t-1})$$

tells us the probability of the next character of the sequence $$\boldsymbol{x}_{t}$$ given (that is, having observed) the previous characters of the sequence $$\boldsymbol{x}_{0:t-1} = (\boldsymbol{x}_{0}, \dots, \boldsymbol{x}_{t-1})$$. In general, the next character of a sequence $$\boldsymbol{x}_{t}$$ can depend on all previous characters of the same sequence, $$\boldsymbol{x}_{0:t-1}$$. If we are able to learn the conditional model $$P(\boldsymbol{x}_{t} \mid \boldsymbol{x}_{0:t-1})$$, given a sequence of $$t$$ characters, $$\boldsymbol{x}_{0:t-1}$$, we can sample the next character according to the same conditional probability distribution.

Recall that, given two events (or random variables) $$A$$ and $$B$$, the joint distribution of them is defined as $$P(A, B) = P(A \mid B)P(B) = P(B \mid A)P(A)$$, which gives rise to the Bayes' theorem. This can be easily generalised to multiple variables. More specifically, if you consider $$B$$ to a set of $$N$$ events (rather than just one), that is, $$B=B_1 \cap B_2\cap \dots \cap B_N = B_1, B_2, \dots, B_N$$, then $$P(A, B) = P(A \mid B)P(B)$$ still holds, but we can further decompose it

\begin{align} P(A, B) = P(A, B_1, B_2, \dots, B_N) &= P(A \mid B_1, B_2, \dots, B_N)P(B_1, B_2, \dots, B_N) \\ &= P(A \mid B_1, B_2, \dots, B_N)P(B_1 \mid B_2, \dots, B_N) P(B_2, \dots, B_N) \\ &= \cdots \end{align}

This is called the chain rule (or product rule) of probability. Essentially, we apply the rule $$P(A, B) = P(A \mid B)P(B)$$ recursively.

Analogously, in the paper, the authors apply this chain rule to express the joint distribution $$P(\boldsymbol{x}_{0:T})$$ as a product of conditional probability distributions, that is

$$P(\boldsymbol{x}_{0:T}) = \prod_{t=0}^T P(\boldsymbol{x}_{t}\mid \boldsymbol{x}_{0:t-1})$$

In the case of the LSTM, the vector $$\boldsymbol{h}_t$$ is supposed to keep track of the past. More specifically, in the case of the character-level language model, we train an LSTM-based RNN, so that $$\boldsymbol{h}_t$$ is approximately equal to $$\boldsymbol{x}_{0:t-1}$$, that is, $$\boldsymbol{h}_t \approx \boldsymbol{x}_{0:t-1}$$. Hence, the joint probability distribution of the characters above can be now be approximately defined as a function of the vector $$\boldsymbol{h}_t$$

$$P(\boldsymbol{x}_{0:T}) \approx \prod_{t=0}^T P(\boldsymbol{x}_{t}\mid \boldsymbol{h}_t; \boldsymbol{\theta})$$

where $$\boldsymbol{\theta}$$ are the parameters of the LSTM-based RNN. Note that this is not an equality but an approximation. Intuitively, we train the LSTM so that it learns the interactions of the past characters.

• Thanks for clarifying my doubt and also for editing the question. – Sarthak Mittal May 30 at 15:37