Which approach can I use to generate text based on multiple inputs?

Question

I have a little experience in building various models, but I've never created anything like this, so just wondering if I can be pointed in the right direction.

I want to create (in python) a model which will generate text based on multiple inputs, varying from text input (vectorized) to timestamp and integer inputs.

For example, in the training data, the input might include:

eventType = ShotMade

shotType = 2

homeTeamScore = 2

awayTeamScore = 8

player = JR Smith

assist = George Hill

period = 1

and the output might be (possibly minus the hashtags): JR Smith under the basket for 2! 8-4 CLE. #NBAonBTV #ThisIsWhyWePlay #PlayByPlayEveryDay #NBAFinals

or

JR Smith out here doing #WhateverItTakes to make Cavs fans forgive him. #NBAFinals

Where is the best place to look to get a good knowledge of how to do this?

Answer 1

Generally, text generators work by modeling the joint distribution of the text by its Bayesian forward decomposition

$ \begin{align*} p(w_1, w_2, ..., w_n) &= p(w_1) * p(w_2|w_1) * p(w_3|w_2, w_1) *\ ...\ * p(w_n|\{w_i\}_{i<n})\\ &= \prod_{i=1}^n p(w_i|\{w_k\}_{k<i})\\ \end{align*} $

From a modeling perspective, this looks right up RNN's ally, where you can have a state holding information from $\{w_k\}_{k<i}$ to learn a representation of $w_i$

Now, in your specific case, you're interested in a conditional text-generator, so you are trying to model $p(w_1, w_2, ..., w_n | \{v_j\}_j)$, but this same tactic works.

$ \begin{align*} p(w_1, w_2, ..., w_n| \{v_j\}_j) &= p(w_1|\{v_j\}_j) * p(w_2|w_1, \{v_j\}_j) * p(w_3|w_2, w_1, \{v_j\}_j) *\ ...\ * p(w_n|\{w_i\}_{i<n}, \{v_j\}_j)\\ &= \prod_{i=1}^n p(w_i|\{w_k\}_{k<i}, \{v_j\}_j)\\ \end{align*} $

So, in your RNN or forward-based model, you can use the exact same approach just additionally embed the conditional inputs you have and somehow infuse it into the model (in practice, I have seen this through attention, concatenation, or some other common approach).

My recommendation (depending on the computational power you have) is to take advantage of the recent fad of pre-trained language models. Specifically, ones trained on next word prediction will probably do the job best. A good example is gpt-2, and, if you check out their GitHub, their code is very readable and easy to adjust for adding conditional input in the ways I have described.