3
$\begingroup$

I have a little experience in building various models, but I've never created anything like this, so just wondering if I can be pointed in the right direction.

I want to create (in python) a model which will generate text based on multiple inputs, varying from text input (vectorized) to timestamp and integer inputs.

For example, in the training data, the input might include:

eventType = ShotMade

shotType = 2

homeTeamScore = 2

awayTeamScore = 8

player = JR Smith

assist = George Hill

period = 1

and the output might be (possibly minus the hashtags): JR Smith under the basket for 2! 8-4 CLE. #NBAonBTV #ThisIsWhyWePlay #PlayByPlayEveryDay #NBAFinals

or

JR Smith out here doing #WhateverItTakes to make Cavs fans forgive him. #NBAFinals

Where is the best place to look to get a good knowledge of how to do this?

$\endgroup$
2
$\begingroup$

Generally, text generators work by modeling the joint distribution of the text by its Bayesian forward decomposition

$ \begin{align*} p(w_1, w_2, ..., w_n) &= p(w_1) * p(w_2|w_1) * p(w_3|w_2, w_1) *\ ...\ * p(w_n|\{w_i\}_{i<n})\\ &= \prod_{i=1}^n p(w_i|\{w_k\}_{k<i})\\ \end{align*} $

From a modeling perspective, this looks right up RNN's ally, where you can have a state holding information from $\{w_k\}_{k<i}$ to learn a representation of $w_i$

Now, in your specific case, you're interested in a conditional text-generator, so you are trying to model $p(w_1, w_2, ..., w_n | \{v_j\}_j)$, but this same tactic works.

$ \begin{align*} p(w_1, w_2, ..., w_n| \{v_j\}_j) &= p(w_1|\{v_j\}_j) * p(w_2|w_1, \{v_j\}_j) * p(w_3|w_2, w_1, \{v_j\}_j) *\ ...\ * p(w_n|\{w_i\}_{i<n}, \{v_j\}_j)\\ &= \prod_{i=1}^n p(w_i|\{w_k\}_{k<i}, \{v_j\}_j)\\ \end{align*} $

So, in your RNN or forward-based model, you can use the exact same approach just additionally embed the conditional inputs you have and somehow infuse it into the model (in practice, I have seen this through attention, concatenation, or some other common approach).

My recommendation (depending on the computational power you have) is to take advantage of the recent fad of pre-trained language models. Specifically, ones trained on next word prediction will probably do the job best. A good example is gpt-2, and, if you check out their GitHub, their code is very readable and easy to adjust for adding conditional input in the ways I have described.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.