Now I know this might break some StackExchange rules and I am definitely open for taking the thread down if it does! I am trying to build an AI that can write it's own book and I have no idea where to start or what are the appropriate algorithms and approaches to go with. How should I start and what do I exactly need for such a project?
$\begingroup$
$\endgroup$
3
-
$\begingroup$ Text generation has a long history in computing, and there are plenty of AI areas that might help. This question might be on topic if you explain more about your project goals and where you are stuck. I thnk some details on the nature of the book and its purpose are important, because state of the art language algorithms can generate text with certain stylistic forms, but which is usually vacuous nonsense. Take a look at talktotransformer.com perhaps - is that the sort of text generation you had in mind? $\endgroup$– Neil SlaterDec 7, 2019 at 15:05
-
$\begingroup$ something like that but not quite exactly, I want to do something like feed the AI some history books about a country (this is just an example because I can't disclose the purpose of the AI) and it generates an alternate history for that country $\endgroup$– ThorvaldDec 7, 2019 at 15:14
-
1$\begingroup$ OK, make up some alternative but similar purpose like the one you suggest and add it to your question. I think that what you want is beyond current AI techniques, especially if you expect anything the AI produces to many any kind of sense. So it may help to understand whether your purpose is in the question to really make such an AI (an impossible task for 2020) or to better understand the technologies that could be involved in such a project and why it is hard to do $\endgroup$– Neil SlaterDec 7, 2019 at 15:23
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
2
There have been many methods proposed for text generating, but recurrent network dominates natural language processing with a key component: the perception of time.
Many networks have been tried for text generation, with notable examples such as Markov chain. However RNN have been proven to work the best and is dominating the field of language modelling (text generation).
How text generation works
A neural network that generates text is commonly called a language model. It is trained on large amount of text with labels being the next token. The text generation process uses several random token as the starting phrase and then the network predicts the rest. However the network does not just predicts the most probable word, instead it randomly chooses one of the few most probable token, hence the generating part.
Why RNN work best on language modelling
RNN have a perception of time built into the architecture of teh network. LSTM, a popular RNN variant used, is composed of "memory units" that "remembers" past text, thus the "time" part. RNN process input according to the sequence of time, so the network can naturally understand time, thus the superior performance compared to other networks.
Architecture of language model
A language model consists of the encoder and the decoder. The encoder compresses word one-hot representation to a smaller sized vector representation. The smaller sized representation is then passed through the decoder, which maps the encoding to the words one hot vectors again.
State of the art results for language modelling
Language modelling is an actively researched field in the AI community, and recently the model GPT-2 have achieved a breakthrough in language modelling accuracy, producing almost human like text with a special component added, the attention layer. Attention basically maps the "memory states" of the encoder and feed it as input to the decoder. The data teh model is trained on is also very large, with over 20GB of web scraped data from sites like Reddit.
Limits of language modelling
One limit of language modelling is the size of generated text. As GPU don't have unlimited memory, language model usually limits the input token size to a specific number, padding or trimming to this number. The number is usually 500-1000, which includes a paragraph or two, but not an entire book. You can only generate short paragraphs and essay with language modelling. For long text it is much harder.
Resources to help you get started
GPT-2 open AI blog: https://openai.com/blog/better-language-models/ GPT-2 online interactive site for text generation: https://talktotransformer.com/ How to train and fine tune GPT-2 in python: https://minimaxir.com/2019/09/howto-gpt2/
Hope I can help you
-
$\begingroup$ So what to do when I'm trying to generate large texts that for sure will exceeds 1000 words? should I do it piece by piece? $\endgroup$– ThorvaldDec 7, 2019 at 19:13
-
$\begingroup$ In short, you simply cannot. The training data of the model is just not a piece of continuous long text, so at the end the AI is going to predict the end token and the sentence/paragraph is going to end. $\endgroup$– ClementDec 8, 2019 at 5:27
$\begingroup$
$\endgroup$
3
Recurrent Neural Networks (RNNs) have been applied to generate text. In this blog post you will find a couple of interesting text examples (the author also has made his code available on github), e.g. their Shakespeare-like texts generated by an RNN:
PANDARUS: Alas, I think he shall be come approached and the day When little srain would be attain'd into being never fed, And who is but a chain and subjects of his death, I should not sleep.
Second Senator: They are away this miseries, produced upon my soul, Breaking and strongly should be buried, when I perish The earth and thoughts of many states.
DUKE VINCENTIO: Well, your wit is in the care of side and that.
Second Lord: They would be ruled after this chamber, and my fair nues begun out of the fact, to be conveyed, Whose noble souls I'll have the heart of the wars.
Clown: Come, sir, I will make did behold your worship.
VIOLA: I'll drink it.
As you can see the RNN is able to somewhat mimic the "flow" of the texts it has been trained on but some sentences (like at the very end) do not make much contextual sense.
Moreover, RNNs have been trained to generate other content, e.g. drawing numbers (see here) or creating music (see here).
-
$\begingroup$ is it a good approach to train the RNN on a large amounts of data (sentences, paragraphs...) ? $\endgroup$– ThorvaldDec 7, 2019 at 15:33
-
1$\begingroup$ Well, that depends on a number of things: what you define as "large amount", your computing power, the amount of data needed to learn the respective task etc. In the linked blog post the author used up to 100MB to train his Wikipedia article writing RNN. Using a rough ball park figure 100MB of pure text equals about 100 books with 500 pages each. $\endgroup$– JonathanDec 7, 2019 at 15:44
-
$\begingroup$ okay, I got you! so the best approach here is using RNNs and go there! Thank you for your time! $\endgroup$– ThorvaldDec 7, 2019 at 16:07